Semantic framework for variable haptic output

ABSTRACT

Methods and apparatus organize a plurality of haptic output variations into a cohesive semantic framework that uses various information about the alert condition and trigger, application context, and other conditions to provide a system of haptic outputs that share characteristics between related events. In some embodiments, an event class or application class provides the basis for a corresponding haptic output. In some embodiments, whether an alert-salience setting is on provides the basis for adding an increased salience haptic output to the standard haptic output for the alert. In some embodiments, consistent haptics provide for branding of the associated application class, application, and/or context.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/835,708 filed Aug. 26, 2015, which claims the benefit of U.S.Provisional Application No. 62/172,122, filed Jun. 7, 2015, and of U.S.Provisional Application No. 62/129,905, filed Mar. 8, 2015, and of U.S.Provisional Application No. 62/044,870, filed Sep. 2, 2014, all of whichare incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates generally to methods and apparatus forproviding haptic feedback, and more particularly relates to methods andapparatus for a semantic framework of haptic feedback that correspondsto conditions associated with a device or application.

BACKGROUND

Tactile feedback systems provide a user with the ability to interactwith a subsystem through touch or contact. Haptic systems facilitatethese tactile interactions by using actuators, sensors, or both.

SUMMARY

The embodiments herein disclose haptic feedback methods and apparatusthat organize a plurality of haptic output variations into a cohesivesemantic framework that uses various information about the alertcondition and triggers, application context, and other conditions toprovide a system of haptic outputs that shares characteristics betweenrelated events. This disclosure relates to systems and methods forproviding haptic response in various conditions. The disclosed hapticresponse approaches may be implemented using any suitable software,hardware, or both.

In some embodiments, a state associated with an application at a timeassociated with a detected alert condition associated with theapplication provides the basis for providing a corresponding hapticoutput. For example, responsive to detecting the alert condition, astate associated with the application at a time associated with thealert condition is determined. If the application was in an active stateat the time associated with the alert condition, a first haptic outputis provided that represents the occurrence of the alert condition and ifthe application was in an inactive state at the time associated with thealert condition, a second haptic output is provided that represents theoccurrence of the alert condition and is different from the first hapticoutput.

In some embodiments, whether a detected alert condition was manually orautomatically triggered provides the basis for providing a correspondinghaptic output. For example, responsive to detecting the alert condition,whether the alert condition was triggered by a manually-initiated eventis determined. If the alert condition was triggered by amanually-initiated event, a first haptic output is providedcorresponding to a manually-initiated event notification. If the alertcondition was triggered by an automatically-initiated event, a secondhaptic output is provided corresponding to an automatically-initiatedevent notification and is different from the first haptic output.

In some embodiments, whether a detected alert condition is associatedwith user input or a predetermined system event provides the basis forproviding a corresponding haptic output. For example, responsive todetecting a first alert condition associated with receiving user inputto an application, a first haptic output is provided corresponding tothe user input. Following the first haptic output, in response todetecting a second alert condition associated with receiving apredetermined system event in the application, a second haptic output ofa greater intensity than the first haptic output is providedcorresponding to the predetermined system event.

In some embodiments, whether a detected alert condition is part of amulti-part operation provides the basis for providing a correspondinghaptic output. For example, responsive to receiving an inputcorresponding to a first part of a multi-part operation, an ongoinghaptic output sequence is initiated. After initiation and responsive toreceiving an input corresponding to a second part of the multi-partoperation, the ongoing haptic output sequence is terminated.

In some embodiments, whether a detected request to perform an operationis a subset of another operation provides the basis for providing acorresponding haptic output. For example, responsive to detecting afirst input that corresponds to a request to perform a first operation,a first output is provided that includes a haptic component and thefirst operation is performed. After performing the first operation, andresponsive to detecting a second input that corresponds to a request toperform a second operation that includes the first operation and asupplemental operation, a second output is provided that includes ahaptic component and includes the first output plus a supplementaloutput that corresponds to the supplemental operation, and the secondoperation is performed.

In some embodiments, whether two detected alert conditions are in thesame or different class of alert condition or class of applicationprovides the basis for providing a corresponding haptic output. Forexample, responsive to detecting the occurrence of a first alertcondition, a first output is provided that includes a first hapticcomponent and a first non-haptic component. After providing the firstoutput, and responsive to detecting the occurrence of a second alertcondition, a second output is provided that includes a second hapticcomponent and a second non-haptic component. If the first alertcondition and the second alert condition are different alert conditionsthat are in a same class of alert condition, the first output and thesecond output share one or more same components and have one or moredifferent components, but if the first alert condition and the secondalert condition are different alert conditions that are in differentclasses of alert condition, the first haptic component is different fromthe second haptic component and the first non-haptic component isdifferent from the second non-haptic component.

Note that the various embodiments of the methods described herein can becombined with any other embodiments described herein. The features andadvantages described in the specification are not all inclusive and, inparticular, many additional features and advantages will be apparent toone of ordinary skill in the art in view of the drawings, specification,and claims. Moreover, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on a portable multifunction device in accordance with someembodiments.

FIG. 4B illustrates an exemplary user interface for a multifunctiondevice with a touch-sensitive surface that is separate from the displayin accordance with some embodiments.

FIG. 5 depicts examples of various haptic waveform morphologiesaccording to some embodiments.

FIG. 6 depicts examples of various audio waveform morphologies accordingto some embodiments.

FIG. 7 is a flow diagram of a method for detecting an alert conditionassociated with an application and providing a corresponding hapticoutput according to some embodiments.

FIG. 8 displays an exemplary user interface of an email applicationaccording to some embodiments.

FIG. 9 is a flow diagram of a method for detecting an alert conditionassociated with an application and providing a haptic output depended onwhether the condition was triggered by a manually- orautomatically-initiated event according to some embodiments.

FIG. 10 is a flow diagram of a method for detecting a first alertcondition from user input and a second alert condition from apredetermined event associated with an application and providingcorresponding haptic outputs according to some embodiments.

FIG. 11 is a flow diagram of a method for detecting an alert conditionassociated with a multi-part operation and providing a correspondinghaptic output according to some embodiments.

FIG. 12 is a flow diagram of a method for detecting first and secondinputs to perform first and second operations, performing theoperations, and providing corresponding outputs according to someembodiments.

FIG. 13 is a flow diagram of a method for detecting first and secondalert conditions and providing corresponding outputs according to someembodiments.

FIG. 14 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 15 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 16 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 17 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 18 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 19 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 20 depicts a device housing including a first acoustic portaccording to some embodiments.

FIG. 21A depicts a touch sensor circuit corresponding to aself-capacitance touch pixel electrode and sensing circuit according tosome embodiments.

FIG. 21B depicts a self-capacitance touch sensor according to someembodiments.

FIG. 22A depicts a haptic actuator according to some embodiments.

FIG. 22B depicts a haptic actuator according to some embodiments.

FIG. 23A is a wearable device according to some embodiments.

FIG. 23B depicts a user wearing a wearable device with a secondelectronic device in his pocket according to some embodiments.

FIG. 24 depicts a schematic diagram of a wearable electronic deviceaccording to some embodiments.

FIG. 25 is a flow diagram of a method for providing various haptic andother feedback alerts according to class according to some embodiments.

FIG. 26 is a flow diagram of a method for providing selective salienceincrease for various haptics according to some embodiments.

FIG. 27 illustrates an exemplary user interface for an alert-saliencesetting on a device in accordance with some embodiments.

FIG. 28 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 29 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 30 is a flow diagram of a method for detecting a first conditionand a second condition and generating corresponding alerts includinghaptic and audio outputs according to some embodiments.

FIG. 31 depicts various instances of a waveform according to someembodiments.

FIG. 32 is a flow diagram of a method for detecting a first condition ina first and second context and generating corresponding alerts includinghaptic and audio outputs according to some embodiments.

FIG. 33A shows various haptic and audio waveforms, both separate andconcurrent in time according to some embodiments.

FIG. 33B shows various haptic and audio waveforms, overlapping in time,according to some embodiments.

FIG. 33C shows various haptic and audio waveforms, overlapping in time,according to some embodiments.

FIG. 34 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 35 is a functional block diagram of an electronic device inaccordance with some embodiments.

DETAILED DESCRIPTION

The embodiments herein disclose haptic feedback methods and apparatusthat organize a plurality of haptic output variations into a cohesivesemantic framework that uses various information about the alertcondition and trigger, application context, and other conditionscorresponding to the haptic output to provide a system of haptic outputsthat shares characteristics between related events.

In some embodiments, a state associated with an application at a timeassociated with a detected alert condition associated with theapplication provides the basis for providing a corresponding hapticoutput. In some embodiments, whether a detected alert condition wasmanually or automatically triggered provides the basis for providing acorresponding haptic output. In some embodiments, whether a detectedalert condition is associated with user input or a predetermined systemevent provides the basis for providing a corresponding haptic output.

In some embodiments, whether a detected alert condition is part of amulti-part operation provides the basis for providing a correspondinghaptic output. In some embodiments, whether a detected request toperform an operation is a subset of another operation provides the basisfor providing a corresponding haptic output. In some embodiments,whether two detected alert conditions are in the same or different classof alert condition provides the basis for providing a correspondinghaptic output.

Exemplary Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Exemplary embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch screen displays and/or touch pads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch screendisplay and/or a touch pad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a drawing application, a presentationapplication, a word processing application, a website creationapplication, a disk authoring application, a spreadsheet application, agaming application, a telephone application, a video conferencingapplication, an e-mail application, an instant messaging application, aworkout support application, a photo management application, a digitalcamera application, a digital video camera application, a web browsingapplication, a digital music player application, and/or a digital videoplayer application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive displays 112 inaccordance with some embodiments. Touch-sensitive display 112 issometimes called a “touch screen” for convenience, and is sometimesknown as or called a touch-sensitive display system. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPU's) 120, peripherals interface 118, RF circuitry 108, audiocircuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem106, other input or control devices 116, and external port 124. Device100 optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensity of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100optionally includes one or more tactile output generators 167 forgenerating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact) on the touchsensitive surface, or to a substitute (proxy) for the force or pressureof a contact on the touch sensitive surface. The intensity of a contacthas a range of values that includes at least four distinct values andmore typically includes hundreds of distinct values (e.g., at least256). Intensity of a contact is, optionally, determined (or measured)using various approaches and various sensors or combinations of sensors.For example, one or more force sensors underneath or adjacent to thetouch-sensitive surface are, optionally, used to measure force atvarious points on the touch-sensitive surface. In some implementations,force measurements from multiple force sensors are combined (e.g., aweighted average) to determine an estimated force of a contact.Similarly, a pressure-sensitive tip of a stylus is, optionally, used todetermine a pressure of the stylus on the touch-sensitive surface.Alternatively, the size of the contact area detected on thetouch-sensitive surface and/or changes thereto, the capacitance of thetouch-sensitive surface proximate to the contact and/or changes thereto,and/or the resistance of the touch-sensitive surface proximate to thecontact and/or changes thereto are, optionally, used as a substitute forthe force or pressure of the contact on the touch-sensitive surface. Insome implementations, the substitute measurements for contact force orpressure are used directly to determine whether an intensity thresholdhas been exceeded (e.g., the intensity threshold is described in unitscorresponding to the substitute measurements). In some implementations,the substitute measurements for contact force or pressure are convertedto an estimated force or pressure and the estimated force or pressure isused to determine whether an intensity threshold has been exceeded(e.g., the intensity threshold is a pressure threshold measured in unitsof pressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be accessible by the user on a reduced-size device withlimited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU 120 and the peripherals interface118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some embodiments, they are, optionally, implemented onseparate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), Audio circuitry 110,speaker 111, and microphone 113 provide an audio interface between auser and device 100. Audio circuitry 110 receives audio data fromperipherals interface 118, converts the audio data to an electricalsignal, and transmits the electrical signal to speaker 111. Speaker 111converts the electrical signal to human-audible sound waves. Audiocircuitry 110 also receives electrical signals converted by microphone113 from sound waves. Audio circuitry 110 converts the electrical signalto audio data and transmits the audio data to peripherals interface 118for processing. Audio data is, optionally, retrieved from and/ortransmitted to memory 102 and/or RF circuitry 108 by peripheralsinterface 118. In some embodiments, audio circuitry 110 also includes aheadset jack (e.g., 212, FIG. 2). The headset jack provides an interfacebetween audio circuitry 110 and removable audio input/outputperipherals, such as output-only headphones or a headset with bothoutput (e.g., a headphone for one or both ears) and input (e.g., amicrophone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch screen 112 and other input control devices 116, to peripheralsinterface 118. I/O subsystem 106 optionally includes display controller156, optical sensor controller 158, intensity sensor controller 159,haptic feedback controller 161 and one or more input controllers 160 forother input or control devices. The one or more input controllers 160receive/send electrical signals from/to other input or control devices116. The other input control devices 116 optionally include physicalbuttons (e.g., push buttons, rocker buttons, etc.), dials, sliderswitches, joysticks, click wheels, and so forth. In some alternateembodiments, input controller(s) 160 are, optionally, coupled to any (ornone) of the following: a keyboard, infrared port, USB port, and apointer device such as a mouse. The one or more buttons (e.g., 208, FIG.2) optionally include an up/down button for volume control of speaker111 and/or microphone 113. The one or more buttons optionally include apush button (e.g., 206, FIG. 2).

Touch-sensitive display 112 provides an input interface and an outputinterface between the device and a user. Display controller 156 receivesand/or sends electrical signals from/to touch screen 112. Touch screen112 displays visual output to the user. The visual output optionallyincludes graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output corresponds to user-interface objects.

Touch screen 112 has a touch-sensitive surface, sensor or set of sensorsthat accepts input from the user based on haptic and/or tactile contact.Touch screen 112 and display controller 156 (along with any associatedmodules and/or sets of instructions in memory 102) detect contact (andany movement or breaking of the contact) on touch screen 112 andconverts the detected contact into interaction with user-interfaceobjects (e.g., one or more soft keys, icons, web pages or images) thatare displayed on touch screen 112. In an exemplary embodiment, a pointof contact between touch screen 112 and the user corresponds to a fingerof the user.

Touch screen 112 optionally uses LCD (liquid crystal display)technology, LPD (light emitting polymer display) technology, or LED(light emitting diode) technology, although other display technologiesare used in some embodiments. Touch screen 112 and display controller156 optionally detect contact and any movement or breaking thereof usingany of a plurality of touch sensing technologies now known or laterdeveloped, including but not limited to capacitive, resistive, infrared,and surface acoustic wave technologies, as well as other proximitysensor arrays or other elements for determining one or more points ofcontact with touch screen 112. In an exemplary embodiment, projectedmutual capacitance sensing technology is used, such as that found in theiPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, Calif.

Touch screen 112 optionally has a video resolution in excess of 100 dpi.In some embodiments, the touch screen has a video resolution ofapproximately 160 dpi. The user optionally makes contact with touchscreen 112 using any suitable object or appendage, such as a stylus, afinger, and so forth. In some embodiments, the user interface isdesigned to work primarily with finger-based contacts and gestures,which can be less precise than stylus-based input due to the larger areaof contact of a finger on the touch screen. In some embodiments, thedevice translates the rough finger-based input into a precisepointer/cursor position or command for performing the actions desired bythe user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch screen 112 or anextension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled to optical sensor controller 158in I/O subsystem 106. Optical sensor 164 optionally includescharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor 164 receives light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor 164 optionally capturesstill images or video. In some embodiments, an optical sensor is locatedon the back of device 100, opposite touch screen display 112 on thefront of the device, so that the touch screen display is enabled for useas a viewfinder for still and/or video image acquisition. In someembodiments, another optical sensor is located on the front of thedevice so that the user's image is, optionally, obtained forvideoconferencing while the user views the other video conferenceparticipants on the touch screen display.

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled tointensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor 165 optionally includes one or more piezoresistive strain gauges,capacitive force sensors, electric force sensors, piezoelectric forcesensors, optical force sensors, capacitive touch-sensitive surfaces, orother intensity sensors (e.g., sensors used to measure the force (orpressure) of a contact on a touch-sensitive surface). Contact intensitysensor 165 receives contact intensity information (e.g., pressureinformation or a proxy for pressure information) from the environment.In some embodiments, at least one contact intensity sensor is collocatedwith, or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 112). In some embodiments, at least one contact intensitysensor is located on the back of device 100, opposite touch screendisplay 112 which is located on the front of device 100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled to peripherals interface 118.Alternately, proximity sensor 166 is coupled to input controller 160 inI/O subsystem 106. In some embodiments, the proximity sensor turns offand disables touch screen 112 when the multifunction device is placednear the user's ear (e.g., when the user is making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled tohaptic feedback controller 161 in I/O subsystem 106. Tactile outputgenerator 167 optionally includes one or more electroacoustic devicessuch as speakers or other audio components and/or electromechanicaldevices that convert energy into linear motion such as a motor,solenoid, electroactive polymer, piezoelectric actuator, electrostaticactuator, or other tactile output generating component (e.g., acomponent that converts electrical signals into tactile outputs on thedevice). Contact intensity sensor 165 receives tactile feedbackgeneration instructions from haptic feedback module 133 and generatestactile outputs on device 100 that are capable of being sensed by a userof device 100. In some embodiments, at least one tactile outputgenerator is collocated with, or proximate to, a touch-sensitive surface(e.g., touch-sensitive display system 112) and, optionally, generates atactile output by moving the touch-sensitive surface vertically (e.g.,in/out of a surface of device 100) or laterally (e.g., back and forth inthe same plane as a surface of device 100). In some embodiments, atleast one tactile output generator sensor is located on the back ofdevice 100, opposite touch screen display 112 which is located on thefront of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled to peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled to an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100 as described in U.S. patent application Ser.No. 11/969,800 filed on Jan. 4, 2008, which is incorporated by referencein its entirety.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, text input module (or setof instructions) 134, Global Positioning System (GPS) module (or set ofinstructions) 135, and applications (or sets of instructions) 136.Furthermore, in some embodiments memory 102 stores device/globalinternal state 157, as shown in FIGS. 1A and 3. Device/global internalstate 157 includes one or more of: active application state, indicatingwhich applications, if any, are currently active; display state,indicating what applications, views or other information occupy variousregions of touch screen display 112; sensor state, including informationobtained from the device's various sensors and input control devices116; and location information concerning the device's location and/orattitude.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VxWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used on iPod (trademark of Apple Inc.) devices.

Contact/motion module 130 optionally detects contact with touch screen112 (in conjunction with display controller 156) and other touchsensitive devices (e.g., a touchpad or physical click wheel).Contact/motion module 130 includes various software components forperforming various operations related to detection of contact, such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts) or to multiple simultaneous contacts (e.g.,“multitouch”/multiple finger contacts). In some embodiments,contact/motion module 130 and display controller 156 detect contact on atouchpad.

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, or the like. In some embodiments, theduration of the contact is used in determining the characteristicintensity (e.g., when the characteristic intensity is an average of theintensity of the contact over time). In some embodiments, thecharacteristic intensity is compared to a set of one or more intensitythresholds to determine whether an operation has been performed by auser. For example, the set of one or more intensity thresholds mayinclude a first intensity threshold and a second intensity threshold. Inthis example, a contact with a characteristic intensity that does notexceed the first threshold results in a first operation, a contact witha characteristic intensity that exceeds the first intensity thresholdand does not exceed the second intensity threshold results in a secondoperation, and a contact with a characteristic intensity that exceedsthe third threshold results in a third operation. In some embodiments, acomparison between the characteristic intensity and one or morethresholds is used to determine whether or not to perform one or moreoperations (e.g., whether to perform a respective option or forgoperforming the respective operation) rather than being used to determinewhether to perform a first operation or a second operation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location, atwhich point the intensity of the contact increases. In this example, thecharacteristic intensity of the contact at the end location may be basedon only a portion of the continuous swipe contact, and not the entireswipe contact (e.g., only the portion of the swipe contact at the endlocation). In some embodiments, a smoothing algorithm may be applied tothe intensities of the swipe contact prior to determining thecharacteristic intensity of the contact. For example, the smoothingalgorithm optionally includes one or more of: an unweightedsliding-average smoothing algorithm, a triangular smoothing algorithm, amedian filter smoothing algorithm, and/or an exponential smoothingalgorithm. In some circumstances, these smoothing algorithms eliminatenarrow spikes or dips in the intensities of the swipe contact forpurposes of determining a characteristic intensity.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event.

Graphics module 132 includes various known software components forrendering and displaying graphics on touch screen 112 or other display,including components for changing the visual impact (e.g., brightness,transparency, saturation, contrast or other visual property) of graphicsthat are displayed. As used herein, the term “graphics” includes anyobject that can be displayed to a user, including without limitationtext, web pages, icons (such as user-interface objects including softkeys), digital images, videos, animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions used by tactile output generator(s) 167 toproduce tactile outputs at one or more locations on device 100 inresponse to user interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, and text input module 134, contactsmodule 137 are, optionally, used to manage an address book or contactlist (e.g., stored in application internal state 192 of contacts module137 in memory 102 or memory 370), including: adding name(s) to theaddress book; deleting name(s) from the address book; associatingtelephone number(s), e-mail address(es), physical address(es) or otherinformation with a name; associating an image with a name; categorizingand sorting names; providing telephone numbers or e-mail addresses toinitiate and/or facilitate communications by telephone 138, videoconference 139, e-mail 140, or IM 141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, contact module130, graphics module 132, and text input module 134, telephone module138 are, optionally, used to enter a sequence of characterscorresponding to a telephone number, access one or more telephonenumbers in address book 137, modify a telephone number that has beenentered, dial a respective telephone number, conduct a conversation anddisconnect or hang up when the conversation is completed. As notedabove, the wireless communication optionally uses any of a plurality ofcommunications standards, protocols and technologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, optical sensor164, optical sensor controller 158, contact module 130, graphics module132, text input module 134, contact list 137, and telephone module 138,videoconferencing module 139 includes executable instructions toinitiate, conduct, and terminate a video conference between a user andone or more other participants in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, e-mail client module 140 includes executable instructions tocreate, send, receive, and manage e-mail in response to userinstructions. In conjunction with image management module 144, e-mailclient module 140 makes it very easy to create and send e-mails withstill or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, the instant messaging module 141 includes executableinstructions to enter a sequence of characters corresponding to aninstant message, to modify previously entered characters, to transmit arespective instant message (for example, using a Short Message Service(SMS) or Multimedia Message Service (MMS) protocol for telephony-basedinstant messages or using XMPP, SIMPLE, or IMPS for Internet-basedinstant messages), to receive instant messages and to view receivedinstant messages. In some embodiments, transmitted and/or receivedinstant messages optionally include graphics, photos, audio files, videofiles and/or other attachments as are supported in a MMS and/or anEnhanced Messaging Service (EMS). As used herein, “instant messaging”refers to both telephony-based messages (e.g., messages sent using SMSor MMS) and Internet-based messages (e.g., messages sent using XMPP,SIMPLE, or IMPS).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, GPS module 135, map module 154, and music player module 146,workout support module 142 includes executable instructions to createworkouts (e.g., with time, distance, and/or calorie burning goals);communicate with workout sensors (sports devices); receive workoutsensor data; calibrate sensors used to monitor a workout; select andplay music for a workout; and display, store and transmit workout data.

In conjunction with touch screen 112, display controller 156, opticalsensor(s) 164, optical sensor controller 158, contact module 130,graphics module 132, and image management module 144, camera module 143includes executable instructions to capture still images or video(including a video stream) and store them into memory 102, modifycharacteristics of a still image or video, or delete a still image orvideo from memory 102.

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, text input module 134, and cameramodule 143, image management module 144 includes executable instructionsto arrange, modify (e.g., edit), or otherwise manipulate, label, delete,present (e.g., in a digital slide show or album), and store still and/orvideo images.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, browser module 147 includes executable instructions tobrowse the Internet in accordance with user instructions, includingsearching, linking to, receiving, and displaying web pages or portionsthereof, as well as attachments and other files linked to web pages.

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, audio circuitry 110, speaker111, RF circuitry 108, text input module 134, e-mail client module 140,and browser module 147, online video module 155 includes instructionsthat allow the user to access, browse, receive (e.g., by streamingand/or download), play back (e.g., on the touch screen or on anexternal, connected display via external port 124), send an e-mail witha link to a particular online video, and otherwise manage online videosin one or more file formats, such as H.264. In some embodiments, instantmessaging module 141, rather than e-mail client module 140, is used tosend a link to a particular online video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, e-mail client module 140, and browser module 147, calendarmodule 148 includes executable instructions to create, display, modify,and store calendars and data associated with calendars (e.g., calendarentries, to do lists, etc.) in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, and browser module 147, widget modules 149 aremini-applications that are, optionally, downloaded and used by a user(e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, and browser module 147, the widget creator module 150 are,optionally, used by a user to create widgets (e.g., turning auser-specified portion of a web page into a widget).

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, and text input module 134,search module 151 includes executable instructions to search for text,music, sound, image, video, and/or other files in memory 102 that matchone or more search criteria (e.g., one or more user-specified searchterms) in accordance with user instructions.

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, audio circuitry 110, speaker111, RF circuitry 108, and browser module 147, video and music playermodule 152 includes executable instructions that allow the user todownload and play back recorded music and other sound files stored inone or more file formats, such as MP3 or AAC files, and executableinstructions to display, present or otherwise play back videos (e.g., ontouch screen 112 or on an external, connected display via external port124). In some embodiments, device 100 optionally includes thefunctionality of an MP3 player, such as an iPod (trademark of AppleInc.).

In conjunction with touch screen 112, display controller 156, contactmodule 130, graphics module 132, and text input module 134, notes module153 includes executable instructions to create and manage notes, to dolists, and the like in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, display systemcontroller 156, contact module 130, graphics module 132, text inputmodule 134, GPS module 135, and browser module 147, map module 154 are,optionally, used to receive, display, modify, and store maps and dataassociated with maps (e.g., driving directions; data on stores and otherpoints of interest at or near a particular location; and otherlocation-based data) in accordance with user instructions.

In conjunction with touch screen 112, display system controller 156,contact module 130, graphics module 132, audio circuitry 110, speaker111, RF circuitry 108, text input module 134, e-mail client module 140,and browser module 147, online video module 155 includes instructionsthat allow the user to access, browse, receive (e.g., by streamingand/or download), play back (e.g., on the touch screen or on anexternal, connected display via external port 124), send an e-mail witha link to a particular online video, and otherwise manage online videosin one or more file formats, such as H.264. In some embodiments, instantmessaging module 141, rather than e-mail client module 140, is used tosend a link to a particular online video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some embodiments, the menubutton is a physical push button or other physical input control deviceinstead of a touchpad.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 137-13, 155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch sensitivedisplay 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display 112, as part of a multi-touchgesture). Peripherals interface 118 transmits information it receivesfrom I/O subsystem 106 or a sensor, such as proximity sensor 166,accelerometer(s) 168, and/or microphone 113 (through audio circuitry110). Information that peripherals interface 118 receives from I/Osubsystem 106 includes information from touch-sensitive display 112 or atouch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In embodiments, peripheralinterface 118 transmits event information only when there is asignificant event (e.g., receiving an input above a predetermined noisethreshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch sensitive display 112 displays more than one view. Views aremade up of controls and other elements that a user can see on thedisplay.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In some embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In some embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. However, insome embodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In some embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay 112, and lift-off of the touch (touch end). In some embodiments,the event also includes information for one or more associated eventhandlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display 112, when a touch is detected on touch-sensitivedisplay 112, event comparator 184 performs a hit test to determine whichof the three user-interface objects is associated with the touch(sub-event). If each displayed object is associated with a respectiveevent handler 190, the event comparator uses the result of the hit testto determine which event handler 190 should be activated. For example,event comparator 184 selects an event handler associated with thesub-event and the object triggering the hit test.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoplayer module 145. In some embodiments, object updater 177 creates andupdates objects used in application 136-1. For example, object updater177 creates a new user-interface object or updates the position of auser-interface object. GUI updater 178 updates the GUI. For example, GUIupdater 178 prepares display information and sends it to graphics module132 for display on a touch- sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In some embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen 112 in accordance with some embodiments. The touch screenoptionally displays one or more graphics within user interface (UI) 200.In this embodiment, as well as others described below, a user is enabledto select one or more of the graphics by making a gesture on thegraphics, for example, with one or more fingers 202 (not drawn to scalein the figure) or one or more styluses 203 (not drawn to scale in thefigure). In some embodiments, selection of one or more graphics occurswhen the user breaks contact with the one or more graphics. In someembodiments, the gesture optionally includes one or more taps, one ormore swipes (from left to right, right to left, upward and/or downward)and/or a rolling of a finger (from right to left, left to right, upwardand/or downward) that has made contact with device 100. In someimplementations or circumstances, inadvertent contact with a graphicdoes not select the graphic. For example, a swipe gesture that sweepsover an application icon optionally does not select the correspondingapplication when the gesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. In someembodiments, the menu button 204 includes a fingerprint sensor thatidentifies a fingerprint on the menu button 204. The fingerprint sensoris optionally used to determine whether a finger on the menu button 204has a fingerprint that matches a fingerprint used to unlock the device100. Alternatively, in some embodiments, the menu button is implementedas a soft key in a GUI displayed on touch screen 112.

In one embodiment, device 100 includes touch screen 112, menu button204, push button 206 for powering the device on/off and locking thedevice, volume adjustment button(s) 208, Subscriber Identity Module(SIM) card slot 210, head set jack 212, and docking/charging externalport 124. Push button 206 is, optionally, used to turn the power on/offon the device by depressing the button and holding the button in thedepressed state for a predefined time interval; to lock the device bydepressing the button and releasing the button before the predefinedtime interval has elapsed; and/or to unlock the device or initiate anunlock process. In an alternative embodiment, device 100 also acceptsverbal input for activation or deactivation of some functions throughmicrophone 113. Device 100 also, optionally, includes one or morecontact intensity sensors 165 for detecting intensity of contacts ontouch screen 112 and/or one or more tactile output generators 167 forgenerating tactile outputs for a user of device 100.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3 are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces (“UI”)that is, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on portable multifunction device 100 in accordance withsome embodiments. Similar user interfaces are, optionally, implementedon device 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

Signal strength indicator(s) 402 for wireless communication(s), such ascellular and Wi-Fi signals; Time 404; Bluetooth indicator 405; Batterystatus indicator 406; Tray 408 with icons for frequently usedapplications, such as: Icon 416 for telephone module 138, labeled“Phone,” which optionally includes an indicator 414 of the number ofmissed calls or voicemail messages; Icon 418 for e-mail client module140, labeled “Mail,” which optionally includes an indicator 410 of thenumber of unread e-mails; Icon 420 for browser module 147, labeled“Browser;” and Icon 422 for video and music player module 152, alsoreferred to as iPod (trademark of Apple Inc.) module 152, labeled“iPod;” and Icons for other applications, such as: Icon 424 for IMmodule 141, labeled “Text;” Icon 426 for calendar module 148, labeled“Calendar;” Icon 428 for image management module 144, labeled “Photos;”Icon 430 for camera module 143, labeled “Camera;” Icon 432 for onlinevideo module 155, labeled “Online Video”; Icon 434 for stocks widget149-2, labeled “Stocks;” Icon 436 for map module 154, labeled “Map;”Icon 438 for weather widget 149-1, labeled “Weather;” Icon 440 for alarmclock widget 149-4, labeled “Clock;” Icon 442 for workout support module142, labeled “Workout Support;” Icon 444 for notes module 153, labeled“Notes;” and Icon 446 for a settings application or module, whichprovides access to settings for device 100 and its various applications136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely exemplary. For example, icon 422 for video and music playermodule 152 is labeled “Music” or “Music Player.” Other labels are,optionally, used for various application icons. In some embodiments, alabel for a respective application icon includes a name of anapplication corresponding to the respective application icon. In someembodiments, a label for a particular application icon is distinct froma name of an application corresponding to the particular applicationicon.

FIG. 4B illustrates an exemplary user interface on a device (e.g.,device 300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tabletor touchpad 355, FIG. 3) that is separate from the display 450 (e.g.,touch screen display 112). Device 300 also, optionally, includes one ormore contact intensity sensors (e.g., one or more of sensors 357) fordetecting intensity of contacts on touch-sensitive surface 451 and/orone or more tactile output generators 359 for generating tactile outputsfor a user of device 300.

Although some of the examples which follow will be given with referenceto inputs on touch screen display 112 (where the touch sensitive surfaceand the display are combined), in some embodiments, the device detectsinputs on a touch-sensitive surface that is separate from the display,as shown in FIG. 4B. In some embodiments the touch sensitive surface(e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) thatcorresponds to a primary axis (e.g., 453 in FIG. 4B) on the display(e.g., 450). In accordance with some embodiments, the device detectscontacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface451 at locations that correspond to respective locations on the display(e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470).In this way, user inputs (e.g., contacts 460 and 462, and movementsthereof) detected by the device on the touch-sensitive surface (e.g.,451 in FIG. 4B) are used by the device to manipulate the user interfaceon the display (e.g., 450 in FIG. 4B) of the multifunction device whenthe touch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse based input or stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B)while the cursor is over a particular user interface element (e.g., abutton, window, slider or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations that include a touch-screen display(e.g., touch-sensitive display system 112 in FIG. 1A or touch screen 112in FIG. 4A) that enables direct interaction with user interface elementson the touch-screen display, a detected contact on the touch-screen actsas a “focus selector,” so that when an input (e.g., a press input by thecontact) is detected on the touch-screen display at a location of aparticular user interface element (e.g., a button, window, slider orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementationsfocus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch-screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch-screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

Haptic Output and Associated Processes

Attention is now directed towards embodiments of haptic output andassociated processes that may be implemented on an electronic device,such as device 300 or portable multifunction device 100. Although thedescription below is with respect to device 100, the embodiments hereincan be incorporated in device 300.

FIG. 5 depicts examples of various haptic waveform morphologiesaccording to some embodiments. The various haptic types serve as atomichaptic components that are used in various combinations to createvarious haptic waveforms having distinctive meanings. The examples beloware described to establish a vocabulary for distinguishing betweenhaptic output types herein. The haptic output types described herein arenot meant to be exhaustive; other haptic waveforms can be used.

Waveforms 505 represent examples haptic waveforms with relatively largeamplitudes, or “tap” type haptic outputs, according to some embodiments,that are readily perceptible (e.g., significantly above the absolutethreshold of touch perceptibility for a given haptic device) by users ofa device outputting the haptic, even if just a singular tap. As seen bythe various tap haptic outputs 505 a-e, the waveforms vary in amplitude,duration, and frequency. Typically, humans can perceive frequencies from5 Hz to 29 Hz, and amplitudes of 0N to 1.0N (sones; equivalent to ˜0-40dBA), depending on kHz. Human hearing of audible amplitudes is the mostsensitive at 1-4 kHz.

Waveforms 510 represent examples of haptic waveforms with relativelysmaller amplitudes (and, optionally, shorter durations) as compared towaveforms 505, or “microtap” type haptic outputs, according to someembodiments, which are less readily perceptible (e.g., near the lowthreshold of touch perceptibility for a given haptic device) for usersof a device outputting the haptic to detect as a singular microtap, butare easily perceptible if multiple microtaps are output in succession.The microtap waveforms 510 a-c also vary in amplitude, duration, andfrequency.

Waveform 515 represents an example combination haptic of a tap followedby two microtaps. Since microtaps less easily perceptible alone,waveforms that have a tap preceding a microtap can “prime” the user bygetting his or her attention with tap, making the user more readilyperceive the microtap.

Waveforms 520 represent examples of “fade” type waveforms, which havewaves for a longer duration that decrease in amplitude over time.Waveform 520 a is a “tap-fade” starting at a greater, tap-levelamplitude, and ending in a microtap-level amplitude, and waveform 520 bis a microtap-fade, starting at a lesser, microtap-level amplitude andending at a lower amplitude.

Waveform 525 represents an example a “buzz” type waveform which iscontinuous high frequency typically lasting for a longer durationoverall (e.g., 0.5 s to 1 s). A buzz can have a higher or loweramplitude according to various embodiments, and is perceptible to theuser as a consistent vibration.

Various audio waveforms optionally are output synchronously to varioushaptic waveforms, so as to correspond in the audio domain to thefrequency, amplitude and duration characteristics of the haptic outputsor to differentiate therefrom. Various atomic haptic components, such asthose described above, are used in various combinations to createvarious haptic waveforms that correspond to audio waveforms. The atomichaptic components are modulated with different parameters (e.g.,amplitudes, count/repetition, timing offsets) to generate differentaudio patterns directly from the haptic device itself, so that the audiowaveform corresponds to the haptic waveform. In addition, some waveformsproduced by haptic actuators (e.g., tactile output generator(s) 167)produce audible output of machine-like sounds such as “ticking”according to some embodiments. Alternatively, in some embodiments, aseparate audio signal generator and output device (e.g., a tonegenerator and speaker) can be used to synchronously output audiowaveforms that have differentiated waveforms from the haptic waveforms.

FIG. 6 depicts examples of various audio waveform morphologies accordingto some embodiments. Waveforms 605 represent examples of audio waveformsof various amplitude, duration, and frequency. Sometimes specializedwaveforms are created to mimic common sounds that correlate with theactivity the audio is to be paired with. Waveforms 610 are examples ofspecialized waveforms. Waveform 610 a is a waveform that mimics awalkie-talkie sound, and waveform 610 b is a waveform that mimics aheartbeat sound.

In some embodiments, when haptic and audio waveforms have correspondingmorphologies, e.g., if the haptic waveform and the audio waveform havepeaks at similar times and of similar durations such that the twowaveforms look similar to each other, they are output synchronously.Waveforms 615 (haptic; dashed line) and 620 (audio; solid line) areshown separately, and then aligned and synchronized in time. In thisexample, the time alignment synchronizes a haptic waveform with a seriesof taps with a heartbeat-sounding audio waveform, to create a joinsounds and feel of a heartbeat, e.g., for use in conjunction with aheartbeat monitor.

In some embodiments, the audio components share attributes, but stilldiffer, e.g., a same ringtone or musical notes (same amplitude,frequency, and duration) played in different pitches or with usingdifferent musical instruments are used. Overlaying different audio(e.g., high vs. low pitch, same audio played in metallic vs. glass vs.ceramic tones, etc.) on the same haptic output creates differentsemantic components. These examples are not meant to be exhaustive;other audio waveforms can be used.

For example, a respective haptic output may be accompanied by acorresponding audio output, with at least a portion of the respectiveaudio output concurrent with at least a portion of the respective hapticoutput, or the respective audio output occurs close in time to therespective haptic output so that the respective haptic output and therespective audio output are perceptually concurrent or simultaneous. Thehaptic and audio waveforms need not be perfectly aligned, and the devicecan take into account the fact that for certain classes of haptic andaudio outputs, the haptic and audio outputs will be perceived by a userto be concurrent or simultaneous even if they are slightly offset intime (e.g., because audio output is processed more quickly than hapticoutput, so providing haptic output before providing audio output will,in some circumstance, cause a user to perceive the audio and hapticoutput as occurring concurrently or simultaneously). Variations in theaccompanying audio portion also can vary between embodiments thatinclude an audio output. For example, each of an amplitude of an audiocomponent associated with the first output, a duration of the audiocomponent associated with the first output, one or more musicalqualities (e.g., pitch, tone, etc.) associated with the audio componentassociated with the first output, can vary in certain circumstances.

Application State-Based Haptic Output

One or more states associated with an application at a time associatedwith a detected alert condition for the application provides the basisfor providing a corresponding haptic output, according to someembodiments. Providing haptic output that correlates with the state ofan application creates a more efficient human-machine interface, therebyreducing the amount of time it takes for a user to perform operations,which consequently reduces energy use and increases battery life forbattery powered devices. FIG. 7 is a flow diagram of a method 700 fordetecting an alert condition associated with an application andproviding a corresponding haptic output according to some embodiments.Note that in some embodiments, different steps may be performed otherthan those shown in FIG. 7.

The method begins by a device 100 detecting 705 an alert conditionassociated with an application executing on a computing device 100according to some embodiments. For an application executing on thecomputing device 100, the alert condition takes various form accordingto various embodiments. An alert condition is any event, notification,or other alert directed to a user of the device. One type of alertcondition corresponds to an event that is automatically triggered by theapplication, or triggered from within the application. For example, analert condition triggered by an application includes predetermined orscheduled alerts, such as reminders, scheduled notifications, or alarmsgenerated by the application. A second type of alert condition is anautomatically initiated event notification that is received by theapplication from a source external to the device. For example, a systemgenerated email or message, such as a mass email, spam, system-generatedemail directed to a mailing list, or any other system generatedcommunication. A third type of alert condition is a manually-initiatedevent notification, received by the application from a human user otherthan the user operating the device. For example, an incoming message orcall from a known email address or phone number on the user's contactlist. Various alert conditions can be detected, e.g., detecting receiptof an email or text message while a user interface of an email ormessaging application is displayed on the display of the device.

Responsive to the alert condition, the device 100 determines 710 a stateassociated with the application at a time associated with the alertcondition, according to some embodiments. In one example, the timeassociated with the alert condition is the time the alert is received.In a second example, the time associated with the alert condition is thetime the alert is detected.

In some embodiments, determining 710 the state associated with theapplication at the time associated with the alert condition comprisesdetermining whether a user interface for the application was displayedon the device at or around the time of the alert condition. In somecircumstances it is the time of the alert condition, and in othercircumstances the time period at issue includes a specified timeinterval surrounding the time associated with the alert condition, forexample during a predefined time window spanning an interval beforeand/or after the time associated with the alert condition.

According to some embodiments, the state can be determined 710 by basedon whether the application was executing on the computing device at thetime associated with the alert condition, with a determination that theapplication was in the active state based on the application executingin the foreground. For example, a user interface element visible on thedisplay of the device while the application is executing. In this case,a determination that the application was in the inactive state based onthe application not executing on the computing device or executing inthe background. For example, the device was not actively in use, or notperforming calculations at the direction of the application, or notdisplaying a user interface element indicating an active state of theapplication. A combination of both the status of the user interfacedisplay and the activity state of the application optionally is used inthe determination 710.

Referring to FIG. 8, there is shown an example of an email application800 that is actively executing in the foreground on the device 100 andfor which the user interface is visible, according to some embodiments.In this example, the email application 800 is in the active state.Conversely, an example of an inactive email application on device 100 isshown in FIG. 4A, in which the email application 800 either is notexecuting or is executing in the background, since it is not visible onthe device 100.

Referring again to FIG. 7, in this example, the state associated withthe application at the time associated with the alert condition isdetermined 710 based on whether the respective user interface windowcorresponding to the application was displayed in the foreground of themulti-application-window user interface at the time associated with thealert condition. For example, if multiple application windows aredisplayed in a stack, the application in the front most position in thez-layer is considered the active application. As known in the art, therespective user interface window is the application window that isreceptive to, and/or configured to receive, user input or userinteraction, sometimes referred to as the window that has focus.

According to some embodiments, one or more other user interface windowsare concurrently displayed on a multi-application-window user interfaceon the device at the time associated with the alert condition, in whichcase determining 710 the state associated with the application at thetime associated with the alert condition comprises determining whether arespective user interface window corresponding to the application wasdisplayed in the multi-application-window user interface at the timeassociated with the alert condition.

According to some embodiments, another way of determining 710 the stateassociated with the application at the time associated with the alertcondition comprises determining whether user input to the applicationwas detected at the time associated with the alert condition, with theuser interaction being different from the alert condition. For example,user input in various circumstances includes user touch input, speechinput, or visual/gaze-based input, or any means of receiving user inputto the application. The time associated with the alert in some cases isthe time of the alert condition, and in other circumstances includes aspecified time interval surrounding the time associated with the alertcondition, for example during a predefined time window spanning aninterval before and/or after the time associated with the alertcondition.

In some embodiments in which it is determined 710 that the applicationwas in the active state based on a detected user interaction at the timeassociated with the alert condition, the device 100 further determines alevel of user engagement associated with the user interaction at thetime associated with the alert condition. User engagement in somecircumstances is a spectrum of possible user interactions with theapplication ranging from a lowest level of engagement such as eye gazealone, through medium levels of engagement such as voice or touch alone,up to a greater level of engagement such as touch, voice, and eye gazecombined to focus on the application (e.g., either current level ofengagement, historical level of engagement over a predetermined timeperiod, or a combination of the two).

Determining 710 the state associated with the application at a timeassociated with the alert condition comprises, in some embodiments,determining a state of the computing device at the time associated withthe alert condition. In this example, determining that the applicationwas in the active state comprises determining that the computing devicewas active at the time associated with the alert condition. For example,an active device includes a device that is powered on (e.g., the displayis on) and/or in active use according to some embodiments. Likewise,determining that the application was in the inactive state comprisesdetermining that the computing device was inactive in some way at thetime associated with the alert condition. For example, a powered downdevice, a device in sleep or hibernation mode, a device not in activeuse, a device with the display turned off each correspond to an inactivedevice in various circumstances.

In accordance with a determination that the application was in an activestate at the time associated with the alert condition, the device 100provides 715 a first haptic output having a first set of outputcharacteristics, according to some embodiments.

The first set of characteristics for the first haptic output includesone or more of: an amplitude, a duration, a regularity, a repetitionfrequency, or a variations of atomic haptic features of the first hapticoutput. In some circumstances, the first haptic output is accompanied bya first audio output. For example, a respective haptic output may beaccompanied by a corresponding audio output, with at least a portion ofthe respective audio output concurrent with at least a portion of therespective haptic output, or the respective audio output occurs close intime to the respective haptic output so that the respective hapticoutput and the respective audio output are perceptually concurrent orsimultaneous. The haptic and audio waveforms need not be perfectlyaligned, and the device 100 can take into account the fact that forcertain classes of haptic and audio outputs, the haptic and audiooutputs will be perceived by a user to be concurrent or simultaneouseven if they are slightly offset in time (e.g., because audio output isprocessed more quickly than haptic output, so providing haptic outputbefore providing audio output will, in some circumstance, cause a userto perceive the audio and haptic output as occurring concurrently orsimultaneously). Variations in the accompanying audio portion also canvary between embodiments that include an audio output. For example, eachof an amplitude of an audio component associated with the first output,a duration of the audio component associated with the first output, oneor more musical qualities (e.g., pitch, tone, etc.) associated with theaudio component associated with the first output, can vary in certaincircumstances.

The first and second sets of characteristics correspond to a device typeof the computing device, according to some embodiments. For example, forthe same event, the device 100 optionally provides different hapticoutputs based on type of device (e.g., phone vs. watch vs. laptopcomputer vs. other handheld device). In some embodiments, however, thedifference between the haptic component of the first output and thehaptic component of the second output are maintained without regard tothe device on which they are being performed. In this circumstance, onall device types the same pattern of haptic outputs is provided, but thedifferences in amplitude of the haptic outputs is based on the device onwhich the outputs are being performed.

Variations in the device itself, and thus the specifics of the hapticoutputs, vary according to different embodiments. For example, in somecircumstances the device has a touch-sensitive display. In someembodiments, the first haptic output and the second haptic output areprovided via the touch-sensitive display on the computing device. Inthis example, the device 100 can both received user touch input via thetouch-sensitive display and provide haptic output via the sametouch-sensitive display. This bi-directional touch sensitivity allowsthe device 100 to provide feedback based on received user inputaccording to some embodiments.

As noticed above, the device 100 optionally determines a level of userengagement associated with the user interaction at the time associatedwith the alert condition. In these circumstances, the device 100 furtherdetermines one or more of the first set of output characteristics forthe first haptic output based on the determined level of engagement. Insome embodiments, amplitude, duration, repetition frequency of the audioand/or haptic component may be varied inversely with a measure of userinteraction/engagement with the application. Thus, for higher degrees ofuser interaction, the first set of output characteristics are selectedso as to make the first output have a lower intensity, such that it ismore subtle and less perceptible. Thus, the user is not exposed to anunduly strong signal when the user already has a high degree ofengagement or interaction with the application. If the user is lessengaged, then a higher intensity signal is provided to the user to drawadditional attention to the device or application, according to someembodiments. For example, if the detected user interaction comprises atouch input or touch gesture (and/or user gaze) reflecting an elevatedlevel of user interaction/engagement, then a weaker or less pronouncedoutput is provided to the user, since the user is likely aware of thealert condition by virtue of being more engaged with the application(interacting with the application through sight or touch). If thedetected user interaction comprises a voice input but no accompanyingtouch or gaze, reflecting lower level or measure of engagement with theapplication, then a stronger and more pronounced output optionally isprovided to draw additional attention to the alert condition.

In accordance with a determination that the application was in aninactive state at the time associated with the alert condition, thedevice 100 optionally provides 720 a second haptic output thatrepresents the occurrence of the alert condition, the second hapticoutput having a second set of output characteristics, wherein the secondhaptic output is different from the first haptic output.

According to some embodiments, the second haptic output is different andhas a greater intensity than the first haptic output. For example, thesignal amplitude optionally is larger, the frequency higher, and/or theduration longer than the first haptic output. The second haptic outputhas a greater intensity than the first haptic output in somecircumstances. In some embodiments, the haptic component of the firstoutput has a first set of parameters (e.g., lower amplitude, lessperiodic/regular, lower repetition frequency or count of events, shorterduration of output, less intrusive/perceptible/pronounced), isconstituted of a first set of haptic features or atomic hapticcomponents that are less pronounced (e.g., microtaps) during the activestate of the device than a second set of parameters during the secondstate. Similarly, if accompanied with respective audio components, theaudio component of the first output has a first set of parameters (e.g.,lower amplitude, shorter duration of output, lessintrusive/perceptible/intense audio notes) during the active state and asecond set of parameters during the inactive state. The purpose of thehigher intensity output is to draw elevated user attention to the alertcondition when the application is in the inactive state according tosome embodiments.

As discussed above, the device 100 state optionally is used to determinethe modulation, or output characteristics, of the haptic output. Forexample, when the device is in a power conserving state (e.g., becausethe device battery life has dropped below a predefined threshold, suchas 5%, 10%, 15%, or 20%), haptic outputs that conserve power are usedaccording to some embodiments (e.g., haptic outputs with loweramplitude, shorter duration, and/or lower frequency), even if thosehaptic outputs are less likely to be noticed by the user. When thedevice is not in the power conserving state, default haptic outputs areused in some circumstances even if those haptic outputs do not conservepower (e.g., haptic outputs with relatively higher amplitude, longerduration, and/or higher frequency relative to the haptic outputs thatconserve power). In some embodiments the difference between the hapticcomponent of the first output and the haptic component of the secondoutput is maintained even when power conserving haptic outputs are usedbecause the device is in a power conserving state.

As with the first haptic output, the second haptic output optionally isaccompanied by an audio output, with the waveform of the second hapticoutput generated being based on, mirroring, and synchronized to awaveform of the accompanying audio output. In some embodiments,waveforms of the haptic components are generated from and match with,mimic, mirror, or are synchronized to waveforms of the respective audiocomponents, as discussed in conjunction with FIG. 6.

Various combinations of haptic and audio can be output by the device toprovide different semantic information to the user. As a first example,the first haptic output is not accompanied by audio output but thesecond haptic output is accompanied by an audio output. As a secondexample, the first haptic output is accompanied by a first audio outputand the second haptic output is accompanied by a second audio outputdifferent from the first audio output.

A given haptic waveform can generate different perceptions and hencesemantic information when accompanied by different audio waveforms. Forexample, a haptic component output with a high frequency audio toneresults in different perceptible unit, and hence semantic information ascompared to the same haptic component output with a low frequency audiotone. The high frequency tone may be used to draw elevated userattention to a significant occurrence in the class of events representedby the haptic output, whereas the low frequency tone may be used toindicate a state change in the class of events. For example, a givenhaptic output may be used to indicate receipt of a text message, andaccompanied by either a high frequency tone for an urgent message (orone from a designated sender) or a low frequency tone for a non-urgentmessage or one received from a non-designated sender. Additionally, fortwo related alerts (e.g., an alert corresponding to receiving a textmessage and an alert corresponding to receiving a mail message) havedifferent corresponding audio components with the same cadence and thususe the same haptic component (e.g., the shared haptic componentprovides information that the two alerts correspond to message receiptalerts, while the different audio components provide information as towhich type of message was received).

In some embodiments, the first haptic output is similar, identical, orsubstantially identical to the second haptic output. As one example, thefirst haptic output is accompanied by an audio output and the secondhaptic output is accompanied by the same audio output; in this examplethe first haptic output is different from the second haptic output.Recall the above example of the active email application 800 of FIG. 8and the inactive one of FIG. 4A. Below is a table showing exemplaryalert condition information for an email application 800 and outputscorresponding to the various conditions, states, modes, and conditions.

TABLE 1 EMAIL Primary Alert Application Haptic Audio Condition StateMode Condition Output Output Email Active Mailboxes No new X X receivedmessage Email Active Inbox No new X X received message Email ActiveMessage No new X X received message Email Active Sent No new X Xreceived message Email Active Draft No new X X received message EmailActive Search No new X X received message Email Active Background No newX X received message Email Inactive N/A No new X X received messageEmail Active Mailboxes New Microtap Email received message ReceiveDefault Email Active Inbox New X Email received message Receive DefaultEmail Active Message New Microtap Email received message Receive DefaultEmail Active Sent New Microtap Email received message Receive DefaultEmail Active Draft New Microtap Email received message Receive DefaultEmail Active Search New Microtap Email received message Receive DefaultEmail Active Background New Fade tap Email received message ReceiveDefault Email Inactive N/A New Fade tap Email received message ReceiveDefault

Table 1 shows a variety of possible outputs based on variations of asingle alert condition of an email being received. The first seven rowsshow the application in an active state with no new message, accordingto various modes that the application could be in when the email isreceived: with the application displaying the mailboxes screen,displaying the inbox, displaying a selected message, displaying the sentmessages box, displaying a draft in progress, an email search field, andwhen the application is executing in the background. Since no newmessage is received in the first eight rows, no output (haptic orotherwise) is present.

However, the last eight rows show the condition of a new message beingreceived and the corresponding outputs. As shown, for most of the activemodes, the haptic output is a microtap and the audio output is an emailreceived default audio. However, for the last row in which theapplication is inactive and the second to last row in which theapplication is active but is executing in the background, the hapticoutput is a fade tap and the audio output remains the email receiveddefault. Thus, the audio output is maintained regardless of whether theapplication is active, but the haptic output has a higher intensity (afade tap versus a microtap) when the application is inactive orexecuting in the background. Table 1 provides conditions andcorresponding outputs according to some embodiments, but is merelyexemplary; other combinations of alerts, states, modes, conditions andoutputs are used according to different embodiments.

As one example, at a first point in time, while the application is inthe active state, the device 100 detects 705 a first alert conditionassociated with the application and responsive to detecting the firstalert condition while the first application is in the active state,provides 715 a first output representing the first alert condition. At asecond point in time, while the first application is in an inactivestate, the device 100 detects a second alert condition and responsive todetecting the second alert condition while that the application is inthe inactive state, providing 720 the second haptic output representingthe second alert condition, the second haptic output different from thefirst haptic output.

It should be understood that the particular order in which theoperations in FIG. 7 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) arealso applicable in an analogous manner to method 700 described abovewith respect to FIG. 7. For example, the inputs, alert conditions,applications, states, and haptic outputs described above with referenceto method 700 optionally have one or more of the characteristics of theinputs, alert conditions, applications, states, and haptic outputsdescribed herein with reference to other methods described herein (e.g.,methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200). Forbrevity, these details are not repeated here.

Alert Condition Trigger-Based Haptic Output

Alert conditions can be manually or automatically triggered. Whether adetected alert condition was manually or automatically triggeredprovides the basis for providing a corresponding haptic output,according to some embodiments. Providing haptic output that correlateswith whether an alert condition was manually or automatically triggeredcreates a more efficient human-machine interface, thereby reducing theamount of time it takes for a user to perform operations, whichconsequently reduces energy use and increases battery life for batterypowered devices. FIG. 9 is a flow diagram of a method 900 for detectingan alert condition associated with an application and providing a hapticoutput dependent on whether the condition was triggered by amanually-initiated or automatically-initiated event according to someembodiments. Note that in some embodiments, different steps may beperformed other than those shown in FIG. 9.

The method begins by the device 100 detecting 905 an alert conditionassociated with an application. For an application executing on acomputing device 100, the alert condition takes various forms accordingto various embodiments. An alert condition is any event, notification,or other alert directed to a user of the device. One type of alertcondition corresponds to an event that is triggered by amanually-initiated event, and a second type of alert conditioncorresponds to an event that is triggered by an automatically-initiatedevent. In some circumstances, the device 100 has a touch-sensitivedisplay.

Responsive to detecting the alert condition, the device 100 determines910 whether the alert condition was triggered by a manually-initiatedevent. This step comprises determining whether the alert conditioncorresponds to an event initiated by a human user. Various circumstancescan contribute to determination by the device 100 whether the event wasinitiated by a human user, as further discussed below in conjunctionwith Table 2.

In some embodiments, determining 910 that the alert conditioncharacteristics indicate that the alert condition was triggered by amanually-initiated event comprises determining whether the alertcondition corresponds to an event initiated by a human user other than auser of the computing device. Some exemplary conditions for thisdetermination also are shown in Table 2 as discussed below.

Determining that the alert condition was triggered by amanually-initiated event, in some embodiments, comprises determiningwhether the alert condition corresponds to an input received from ahuman user, identifying a user of the computing device, and comprisingan indication from the human user to alert the user of the computingdevice as to the input. In some embodiments, the input received from ahuman user is from the user of the device 100, however, in someembodiments the input is received from another user on another device(e.g., a sender of a personal message from the other user to the deviceuser).

Table 2 shows a few exemplary circumstances under which the device 100has information used to determine 910 that the alert condition wastriggered by a manually-initiated event, whether it was initiated by ahuman user, and whether that user was a user different from the user ofthe device 100. The applications shown in Table 2 include Messages,Email, Phone, Calendar, Timer, Activity Monitor, and Social Networking,but could be any other applications providing similar information to thedevice 100 for making this determination.

TABLE 2 MANUAL/AUTOMATIC Alert Human/ Appli- Con- Con- Auto Haptic Audiocation dition dition Initiated User Output Output Messages Message NewHuman/ No Tap Message received message Auto Receive Default MessagesMessage New Human Yes Tap- Message sent message Tap Sent Alert EmailEmail New Human/ No X Email received message Auto Receive Default EmailEmail New Human Yes X Email sent message Send Default Phone IncomingCall Human/ No Buzz- Default call Auto Buzz ring Phone Answer AnswerHuman Yes Micro- Default call tap receiver Phone Hang up Hang up HumanYes Micro- Default call tap receiver Calendar Reminder Event Auto No TapDefault start Reminder Timer End Time Auto No Buzz- Default elapsed buzztimer Activity Update Activity Auto No Tap Intense Monitor targetActivity reached Chime Social Notifi- Tagged Human No Tap Default Net-cation in post Notifi- working cation Social Email Activity Auto No XEmail Net- Summary Receive working Default

For the examples shown, the human/auto column shows whether theparticular alert, condition, and application corresponds to an alertthat could be manually (human) initiated, automatically initiated, or ifit could be initiated by either human or automatic means. For example,the second, fourth, sixth, and seventh rows each indicate a manualaction by a human user. In these examples, an action has been takenspecifically by the user of the device (User column), such as user inputinto an application on the device (e.g., the Message, Email, or Phoneapplications), so the device 100 can easily determine that the user washuman user, and the device user more specifically. However, in othercircumstances some alert conditions are of the type known to the device100 as able to be generated either manually, by a human user, orautomatically. For example, rows one, three, and five have alertconditions in this category. Specifically, a New message received in aMessages application typically is a manual (human-initiated) event,e.g., a text or other message sent by a different device user. However,messages also can be auto-generated by a system, such as spam oradvertising text messages. Similarly, a New email message can beinitiated by either a human user or automatically by a system, such as amass email, spam email, or subscription listserv email, which could begenerated by a bot, for example. In the final example, an Incoming callin a Phone application typically is a human-initiated event, but alsocan be an auto-initiated call by a system auto-dialing a list ofcallers, e.g., soliciting donations or as part of a political campaign.In each of these circumstances, additional information is needed aboutthe source of the alert condition to be able to make a more conclusivedetermination as to whether the event was manually or automaticallyinitiated.

For example, for an email or message, the device 100 can determinewhether the alert condition corresponds to a personal communication froma sender in a list of contacts managed, created, controlled, orotherwise associated with the user. The simplest example is an actualcontacts file, such as contacts cards with email addresses or phonenumbers, but other contacts lists also include email addresses or phonenumbers that the user previous has communicated to or from in somecircumstances. These examples are just that; there are many other meansby which the device 100 could ascertain whether a communication islikely to have been initiated manually by a human user.

Another example is shown in the second to last row of Table 2. In thisexample, a Social Networking application sends a Notification that theuser has been Tagged in a post. While the communication itself came froma social networking server, the content of the Notification indicatesthat the post was initiated manually by a human user who is connected tothe device user within the social networking application.

In accordance with a determination that the alert condition wastriggered by a manually-initiated event, the device 100 provides 915 afirst haptic output corresponding to a manually-initiated eventnotification according to some embodiments.

Once the device 100 determines that the alert condition was initiated bya manual event, a haptic output is provided that corresponds to thealert condition. The first haptic output is provided via atouch-sensitive display of the computing device, according to someembodiments. In some embodiments, the determining further comprisesdetermining a measure of urgency associated with the manually-initiatedevent and modulating 912 an intensity of the first haptic output basedon the measure of urgency. For example, the device 100 analyzes thecontent of an incoming message from another end user to determine theurgency of the message. Accordingly, the device 100 examines dateinformation in an email or subject/contents of a message, keywords usedin the subject or message (e.g., “urgent”), or examines metadataincluded in the message such as an “important” flag in some embodiments.The device 100 determines whether senders or recipients of the messageinclude individuals flagged by the user as important people or with someother special designation according to some embodiments. The device 100then provides 915 haptic and/or audio output that is proportionate inintensity (e.g., amplitude, duration, frequency) to the urgencyassociated with the content of the incoming message. In general,expected or less urgent notifications/outputs have one set of outputcharacteristics and unexpected notifications have a second set of outputcharacteristics that are more intense, in order to attempt to drawelevated user attention, according to some embodiments.

In some circumstances, the device 100 determines one or more contextualattributes associated with the manually-initiated event and modulates912 the intensity of the haptic output on that basis. For example,various factors may be considered to evaluate contextual attributesassociated with an incoming manual notification and the strength of thehaptic or audio notification that should be provided. Contextualattributes include an identity of the recipient sending the message,inclusion of and identification of other recipients of the message,timing of receiving the message, location of the user when receiving thenotification, activity being performed by the user when receiving thenotification, the medium used to communicate the message, etc.; thislist is not exhaustive.

According to some embodiments, wherein the first haptic output thatcorresponds to the manually-initiated event notification is more intensethan the second haptic output that corresponds to theautomatically-initiated event notification. For example, the moreintense output is more likely to draw elevated user attention to themanually-initiated event. In some circumstances, by virtue of being morepersonal, targeted, and having a higher likelihood of being directlyrelevant to the user of the device, the haptic for themanually-initiated alert helps to draw the user's immediate attention toan event determined by the system to be more in need of the user'simmediate attention.

For example, the first haptic output is characterized by a first set ofcharacteristics comprising one or more of: an amplitude of the firsthaptic output, a duration of the first haptic output, a regularityassociated with the first haptic output, repetition frequency of hapticfeatures in the first haptic output, and a selection of haptic featuresconstituting the first haptic output, according to some embodiments. Thesecond haptic output is characterized by a second set of characteristicscomprising one or more of: an amplitude of the second haptic output, aduration of the second haptic output, a regularity associated with thesecond haptic output, repetition frequency of haptic features in thesecond haptic output, and a selection of haptic features constitutingthe second haptic output.

In some circumstances two manually-initiated events can be received in arow. A first alert condition can be detected corresponding to anincoming email message from a first human sender and responsive todetermining that the incoming email message corresponds to amanually-initiated event, the device 100 provides the first hapticoutput. A second alert condition is detected corresponding to anincoming text message from a second human sender and responsive todetermining that the incoming text message corresponds to amanually-initiated event, the device 100 provides the first hapticoutput. Many other variations are possible according to someembodiments.

In response to detecting the alert condition associated with theapplication, in accordance with a determination that the alert conditionwas triggered by an automatically-initiated event, the device 100provides 920 a second haptic output corresponding to anautomatically-initiated event notification, wherein the second hapticoutput is different from the first haptic output.

Many examples exist of alert conditions triggered byautomatically-initiated events. Referring again to Table 2, the alertconditions for the Calendar, Timer, and Activity Monitor applicationseach are of the automatic variety, and are generated by the applicationitself. In a slightly different example, the last row shows an alertcondition that is an Activity Summary for social networking in the formof an email. In this example, the alert condition is a scheduled,regular occurrence. Thus, even though the summary may include some eventinformation related to user-initiated actions (e.g., the example aboveof being tagged in a post by another user), the email from the socialnetwork summarizes activity over the course of a day or a week thatincludes the specific post, but the email is not considered amanually-initiated event, because it was not triggered by a request orindication from a human user (it merely includes that information). Thesecond haptic output is provided via a touch-sensitive display of thecomputing device, according to some embodiments.

According to some embodiments, the device 100 determines that the alertcondition was automatically triggered by the application (e.g., apredetermined or scheduled alert such as a reminder notification or anautomated alarm generated by the application), or was anautomatically-initiated event notification received by the applicationfrom a source external to the device (e.g., anautomatically/system-generated email or message). For example, asender's email address of noreply@domain.com is a hint that the emailwas sent by automatic means, as there is no actual human sender to replyto in response to the email.

In accordance with a determination that the automatically-initiatedevent corresponds to an event occurring external to the device 100, thedevice 100 provides a first variant of the second haptic outputaccording to some embodiments, corresponding to a notification of anexternally occurring automatically-initiated event. Likewise, inaccordance with a determination that the automatically-initiated eventcorresponds to an event initiated within the device, the device 100provides a second variant of the second haptic output corresponding to anotification of an internally-generated, automatically-initiated event.Thus, different notifications are provided for the same applicationdepending on whether the event was generated by an incoming message ornot according to some embodiments.

Determining by the device 100 that the alert condition occurs atpredetermined time or reports on satisfaction of a predeterminedtriggering condition indicates that the alert condition has an automatictrigger according to some embodiments. For example, satisfaction of somepredetermined triggering condition (e.g., your activity goal for the dayhas been met, you have left the geo-fenced area, etc.) is anautomatic-type trigger.

As with manually-initiated alert triggers, the device 100 analyzescharacteristics associated with the alert condition to determine one ormore contextual attributes associated with the automatically-initiatedevent and modulates 912 intensity of the second output based on the oneor more contextual attributes. Various factors are considered whenevaluating contextual attributes associated with an automatednotification and the strength of the haptic or audio notification thatshould be provided according to various circumstances. For example, thetype of notification (e.g., a one-time event versus a repeating eventsuch as a periodic/repeating alarm), location of the user at a time ofoccurrence of the event, externally triggered or internally triggerednotifications, activities being performed by the user at a time ofreceiving the notification, and for an external notification, the mediumused to communicate the message.

As with manually-triggered events, for an automatically-initiated eventthe device 100 determines a time-varying measure of urgency associatedwith the automatically-initiated event and modulates 912, over aspecified window of time, a intensity of the second haptic output toindicate the time-varying measure of urgency. For example, for anautomatically-initiated (e.g., internal, prescheduled) trigger, the setof haptic parameters are modulated 912 based on urgency or timing of theevent associated with the notification. For instance, for a cameraapplication, the device 100 provides a series of microtap outputsbuilding up to the photo being taken, the intervals between themicrotaps progressively decreasing as the time for taking the photoapproaches according to some embodiments, providing a time-varyingoutput that builds up the sense of urgency or aids in user understandingof the timing factor.

In some embodiments, the first set of characteristics for the firsthaptic output differs from the second set of characteristics for thesecond haptic output. Similarly, if accompanied by respective audiocomponents, the audio component of the first output has a first set ofparameters (higher amplitude, longer duration of output, moreintrusive/perceptible/pronounced audio notes) for the manually-initiatedevent notification than for the automatically-initiated eventnotification.

A specific example includes detecting an alert condition associated withan application and responsively determining whether the alert conditioncorresponds to a personal communication from a sender in a list ofcontacts associated with the user, in which case a first haptic outputcorresponding to a personal alert notification is provided, or whetherthe alert condition does not correspond to a personal communication froma sender in a list of contacts associated with the user, in which case asecond haptic output corresponding to an automatic alert notification isprovided, with the first haptic output having a greater intensity thanthe second haptic output.

A haptic output optionally is accompanied by a corresponding audiooutput, with at least a portion of the respective audio output beingconcurrent with at least a portion of the respective haptic output, orthe respective audio output occurring close in time to the respectivehaptic output so that the haptic output and the respective audio outputare perceptually concurrent or simultaneous. The haptic and audiowaveforms need not be perfectly aligned, and the device 100 can takeinto account the fact that for certain classes of haptic and audiooutputs, the haptic and audio outputs will be perceived by a user to beconcurrent or simultaneous even if they are slightly offset in time(e.g., because audio output is processed more quickly than hapticoutput, so providing haptic output before providing audio output will,in some circumstance, cause a user to perceive the audio and hapticoutput as occurring concurrently or simultaneously). Variations in theaccompanying audio portion also can vary between embodiments thatinclude an audio output. For example, each of an amplitude of an audiocomponent associated with the first output, a duration of the audiocomponent associated with the first output, one or more musicalqualities (e.g., pitch, tone, etc.) associated with the audio componentassociated with the first output, can vary in certain circumstances.

As with the first haptic output, the second haptic output is accompaniedby an audio output, with the waveform of the second haptic outputgenerated being based on, mirroring, and synchronized to a waveform ofthe accompanying audio output, according to some embodiments. In someembodiments, waveforms of the haptic components are generated from andmatch with, mimic, mirror, or are synchronized to waveforms of therespective audio components, as discussed in conjunction with FIG. 6.

Various combinations of haptic and audio waveforms are output accordingto some embodiments. According to some embodiments, the first hapticoutput is not accompanied by audio output but the second haptic outputis accompanied by an audio output. In one example, the first hapticoutput is accompanied by a first audio output and the second hapticoutput is accompanied by a second audio output; in this case the firsthaptic output is identical to the second haptic output and the firstaudio output is different from the second audio output. In someembodiments, the same haptic waveform generates different perceptionswhen accompanied by different audio waveforms. For example, if an audio(ring) tone is played with a high pitch versus a low pitch, andaccompanied by the same haptic component (or substantially similarhaptic components that a typical user will not be able to differentiatebetween), the perception of the haptic component differs despite theidentical haptic. The higher pitched or shrill version of the ring tonemight be used with the second output to draw elevated user attention. Insome embodiments, the first haptic output is similar, identical, orsubstantially identical to the second haptic output.

In another example the first haptic output is accompanied by an audiooutput and the second haptic output is accompanied by the same audiooutput; in this example the first haptic output is different from thesecond haptic output.

It should be understood that the particular order in which theoperations in FIG. 9 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) arealso applicable in an analogous manner to method 900 described abovewith respect to FIG. 9. For example, the inputs, alert conditions,events, applications, states, and haptic outputs described above withreference to method 900 optionally have one or more of thecharacteristics of the inputs, alert conditions, events, applications,states, and haptic outputs described herein with reference to othermethods described herein (e.g., methods 700, 1000, 1100, 1200, 1300,2500, 2600, 3000, and 3200). For brevity, these details are not repeatedhere.

Feedback Versus Notification-Based Haptic Output

Alert conditions can be triggered by user input or by predeterminedsystem events (e.g., reminders, notification). Whether a detected alertcondition is associated with user input or a predetermined system eventprovides the basis for providing a corresponding haptic output,according to some embodiments. Providing haptic output that correlateswith the type of alert condition creates a more efficient human-machineinterface, thereby reducing the amount of time it takes for a user toperform operations, which consequently reduces energy use and increasesbattery life for battery powered devices. FIG. 10 is a flow diagram of amethod 1000 for detecting a first alert condition from user input and asecond alert condition from a predetermined event associated with anapplication and providing corresponding haptic outputs according to someembodiments. Note that in some embodiments, different steps may beperformed other than those shown in FIG. 10.

The method begins by detecting 1005 a first alert condition on thecomputing device 100 associated with receiving user input to anapplication. The first alert condition can be any of the various alertconditions described elsewhere herein, and the application can be anyapplication executing on the device 100, according to some embodiments.According to some embodiments, the computing device 100 comprises atouch-sensitive display for receiving the user input.

Responsive to detecting the first alert condition, the device 100provides 1010 a first haptic output having a first intensity andcorresponding to the user input to the application, according to someembodiments. The first haptic output optionally is feedback provided indirect response to and as a consequence of the user input. According tosome embodiments, more intense haptics (e.g., higher amplitude or longerhaptics) are used for unexpected alerts that are not responsive to userinput than those haptics used for feedback that is in direct response touser inputs. In some embodiments, unexpected alerts are alerts thatoccur outside of a context where the user's attention is directed towardthe device (e.g., alerts that occur while a user is not actively usingthe device or while the user is not looking at the device). For examplean electronic message alert that is generated by the device while theuser of the device is wearing the device but not looking at the deviceand not interacting with a messaging app is an unexpected alert, whilethe same alert received while the user is looking at the device an/orhas the messaging application open on the device would be an expectedalert. Similarly, feedback for specific user inputs (e.g., a “click”when a user places a pin on a map) is expected by the user because theuser is actively engaged with the device and thus is more likely tonotice a less intense haptic alert. According to some embodiments, thefirst haptic output is provided via the touch-sensitive display on thecomputing device 100. The first intensity can be a pre-determined oradjustable combination of haptic waveform amplitude, frequency, and/orduration, as described elsewhere herein.

The device also 100 detects 1015 a second alert condition associatedwith receiving a predetermined system event in the application. Forexample, the second alert condition corresponds to an event that isautomatically triggered by the application, or from within theapplication. Such events include an alert that occurs at a predeterminedtime or reports on satisfaction of a predetermined criterion, a remindernotification or an alarm generated by the application, anautomatically-initiated event notification received by the application(e.g., an automatically/system generated email or message such as asystem-generated mass mail, spam, system-generated email directed to amailing list, etc.), or a manually initiated event notification receivedby the application (e.g., an event initiated by a human user—such as anincoming message or call from a known email address or phone number onthe user's contact list), according to various embodiments.

Responsive to detecting the second alert condition, the device 100provides 1020 a second haptic output having a second intensity, andcorresponding to the predetermined system event, wherein the secondintensity is greater than the first intensity. The second intensity isgreater than the first intensity in one or more of amplitude, frequency,and/or duration. When the computing device 100 comprises atouch-sensitive display, the second haptic output is provided via thetouch-sensitive display on the computing device.

The greater second intensity of the second haptic output is so designedto draw elevated user attention to alert conditions that do not resultfrom user input. For example, user feedback while using anapplication—in which case a finger may already be touching the device100—requires a lesser haptic output for the user to notice it ascompared to a user who is not touching the device with a finger. Thefeedback while touching the device is expected by the user, and thusoptionally is weaker and more subtle than unexpected output. Directmanipulation should result in short, immediate haptic feedback (<100 ms)according to some embodiments. Haptic feedback is less intrusive (e.g.,has lower amplitude, fewer haptic features, more periodic/repetitive,shorter duration, weaker haptic features, etc.) when it is responsive toa user input. Notifications, on the other hand, often are unexpectedoutput, and thus need to be stronger in order to draw elevated userattention in some circumstances. Thus, notifications optionally arelonger (˜1.5 s) and more intrusive.

As an example, a device 100 detects 1005 a first alert conditioncorresponding to a user selection of a user interface element displayedon an application user interface associated with a first application,and responsive to the first alert condition, provides 1010 a respectivefirst haptic output representing the user selection of a user interfaceelement, the respective first haptic output comprising a first hapticcomponent including a first haptic feature of a first intensity and afirst duration, according to some embodiments. After providing therespective first haptic output, the device 100 detects 1015 a secondalert condition corresponding to an alert notification received by thefirst application, and responsive to the second alert condition,provides 1020 a respective second haptic output representing the receiptof the alert notification, the respective second haptic outputcomprising a second haptic component: (1) including a second hapticfeature of a second intensity and a second duration, and (2) differentfrom and more pronounced than the respective first output, based on thesecond haptic feature being more pronounced than the first hapticfeature, the second intensity being greater than the first intensity, orthe second duration being longer than the first duration according tosome embodiments. For example, an incoming phone call might have a moreintense haptic output than answering or hanging up a call on a phoneapplication.

TABLE 3 PHONE Primary Appli- Alert cation Haptic Audio Condition StateMode Condition Output Output Incoming Active Keypad No call X X callIncoming Active Favorites No call X X call Incoming Active Recents Nocall X X call Incoming Active Contacts No call X X call Incoming ActiveVoicemail No call X X call Incoming Active Call in No call X X callprogress Incoming Active Keypad Call Buzz-Buzz Default call ringIncoming Active Favorites Call Buzz-Buzz Default call ring IncomingActive Recents Call Buzz-Buzz Default call ring Incoming Active ContactsCall Buzz-Buzz Default call ring Incoming Active Voicemail CallBuzz-Buzz Default call ring Incoming Active Call in Call Buzz-BuzzDefault call progress ring Incoming Inactive N/A No call X X callmessage Incoming Inactive N/A Call Buzz-Buzz Default call ring Answercall Active Incoming No answer X X Answer call Active Incoming AnswerMicrotap Default receiver Answer call Inactive Incoming Answer MicrotapDefault receiver Hang up Active Call in No hang up X X call progressHang up Active Call in Hang up Microtap Default call progress receiver

Table 3 shows a phone application and outputs corresponding to threedifferent alert conditions: incoming call, answer call, and hang upcall. In this example, a user answering or hanging up a call an actionthat is expected by the user taking the action, and thus thecorresponding haptic output that accompanies the action is consideredfeedback to the user. For these two alert conditions, the haptic outputis less intense (e.g., a microtap). In contrast, an incoming call is amore unexpected type of alert condition, and in fact one that exists fora limited period of time, and thus has a haptic with a greater intensityintended to get the user's attention (e.g., a buzz-buzz). According tosome embodiments, more intense haptics (e.g., higher amplitude or longerhaptics) are used for unexpected alerts that are not responsive to userinput than those haptics used for feedback that is in direct response touser inputs.

Table 3 provides conditions and corresponding outputs according to someembodiments, but is merely exemplary; other combinations of alerts,states, modes, conditions, and outputs are used according to differentembodiments.

In some circumstances, haptic outputs can vary based on whether theyresult from a specific user input or not. In some embodiments, thedevice 100 first detects a first alert condition associated with a firstapplication, the first alert condition resulting from a respective userinput. For example, a user providing touch inputs to type an emailmessage in an email application. The device 100 provides a respectivefirst haptic output representing the first alert condition according tosome embodiments. Continuing the above example, the device provides lowintensity, short interval buzzes for each user contact input. Afterproviding the respective first haptic output, the device 100 detects asecond alert condition associated with the first application, the secondalert condition not resulting from receiving user input. In the emailapplication context, this alert condition optionally is an incomingemail in email application. The device 100 then provides a respectivesecond haptic output representing the second alert condition, therespective second haptic output different from and having greaterintensity than the respective first output according to someembodiments. For example, the second haptic output optionally has ahigher intensity and/or prolonged taps to notify a user of the device ofthe incoming message.

The first and second haptic outputs can have various characteristicsthat impact their intensity and their ability to draw the attention ofthe user according to some embodiments. For example, a first hapticoutput has a first set of characteristics comprising one or more of: anamplitude of the first haptic output, a duration of the first hapticoutput, a regularity associated with the first haptic output, repetitionfrequency of haptic features in the first haptic output, a selection ofhaptic features constituting the first haptic output and the secondhaptic output has a second set of characteristics comprising one or moreof: an amplitude of the second haptic output, a duration of the hapticcomponent associated with the second haptic output, a regularityassociated with the second haptic output, repetition frequency of hapticfeatures in the second haptic output, a selection of haptic featuresconstituting the second haptic output. Any of the characteristics candiffer between the first and second haptic, or all of them can bedifferent according to various embodiments.

Various combinations of haptic and audio are output according to someembodiments. According to some embodiments, the first haptic output isnot accompanied by audio output but the second haptic output isaccompanied by an audio output. In one example, the first haptic outputis accompanied by a first audio output and the second haptic output isaccompanied by a second audio output; in this case the first hapticoutput is identical to the second haptic output and the first audiooutput is different from the second audio output. In some embodiments,the same haptic waveform generates different perceptions whenaccompanied by different audio waveforms. For example, if an audio(ring) tone is played with a high pitch versus a low pitch, andaccompanied by the same haptic component (or substantially similarhaptic components that a typical user will not be able to differentiatebetween), the perception of the haptic component differs despite theidentical haptic. The higher pitched or shrill version of the ring tonemight be used with the second output to draw elevated user attention. Insome embodiments, the first haptic output is similar, identical, orsubstantially identical to the second haptic output.

Any of the haptic outputs can be accompanied by an audio output, withthe waveform of the second haptic output generated being based on,mirroring, and synchronized to a waveform of the accompanying audiooutput according to some embodiments. In some embodiments, waveforms ofthe haptic components are generated from and match with, mimic, mirror,or are synchronized to waveforms of the respective audio components, asdiscussed in conjunction with FIG. 6. The haptic and audio waveformsneed not be perfectly aligned, and the device 100 can take into accountthe fact that for certain classes of haptic and audio outputs, thehaptic and audio outputs will be perceived by a user to be concurrent orsimultaneous even if they are slightly offset in time (e.g., becauseaudio output is processed more quickly than haptic output, so providinghaptic output before providing audio output will, in some circumstance,cause a user to perceive the audio and haptic output as occurringconcurrently or simultaneously). Variations in the accompanying audioportion also can vary between embodiments that include an audio output.For example, each of an amplitude of an audio component associated withthe first output, a duration of the audio component associated with thefirst output, one or more musical qualities (e.g., pitch, tone, etc.)associated with the audio component associated with the first output,can vary in certain circumstances.

In some embodiments, there is coordination between the haptic intensityand the volume of the device. For example, the first intensity of thefirst haptic output and the second intensity of the second haptic outputis adjusted in inverse relation to a volume setting on the device, suchthat the haptic output optionally is intensified, either independentlyor over and above the default/predetermined intensity settings, if theaudio channel is turned off or turned down.

In some embodiments, prior to providing the first haptic output, thedevice 100 determines 1025 whether an input-based modulation conditionexists, and responsive to determining that an input-based modulationcondition exists, prior to providing the first haptic output, the device100 modifies 1030 the first haptic output.

For example, one input-based modulation condition is user engagement. Inthis case, the device 100 detects a respective alert condition anddetermines that the respective alert condition resulted from arespective user input received on the computing device. Then the device100 identifies a measure of user engagement associated with the detecteduser input and modulates the first signal intensity of the first hapticoutput based on the measure of user engagement according to someembodiments. For example, in some embodiments, the strength (e.g.,amplitude, duration, repetition frequency) of the audio and/or hapticcomponents are varied inversely with a measure of user inputinteraction/engagement with the application/device. For higherdegrees/measures of user interaction/engagement (such as touch or directmanipulation), the output characteristics are selected so as to make thefirst output more subtle or less invasive/perceptible. Thus, the user isnot bombarded with an unduly strong feedback signal when the user has ahigh degree of engagement or interaction with the application/device.For example, weakest output is provided for touch/direct manipulation,e.g., if the user is making contact with the device while providing theinput, then the user should not be bombarded with unduly strong hapticfeedback. On the other hand, if the user is less engaged, then astronger signal is provided to the user to draw additional attention.

TABLE 4 LEVEL OF ENGAGEMENT/HAPTIC FEEDBACK Primary Appli- Alert cationPrimary Secondary Haptic Condition State Focus Focus Condition OutputFeedback Active Application Direct New event Level 1 manipulationNotification Active Application Eye gaze New event Level 2 NotificationActive Other Touch New event Level 2 application Notification ActiveBackground Touch New event Level 2 Notification Active Other No touchNew event Level 3 application Notification Active Background No touchNew event Level 3 Notification Inactive Background No touch New eventLevel 4

Table 4 shows various levels of user engagement for alert conditionswith variations in the user's focus, and the resultant haptic outputlevels. The haptic output levels shown can be thought of as a slidingscale of the level of haptic intensity needed to get the user'sattention in view of the primary and secondary user focus. For example,the highest level of user engagement is direct manipulation of theapplication generating the alert condition, when the alert condition isin the form of feedback. For example, if the user is inputtinginformation into an application, the application is already the user'sprimary focus, so little haptic is necessary to get the user'sattention, and thus the lowest level of haptic, “Level 1,” can beoutput. This may be the most minimal haptic possible, e.g., a singlemicrotap.

Skipping to the third row, if the user it touching the device but isinputting into a different application, the user is focused on thedevice, and thus has some level of engagement, but since the user isusing a different application than the one providing the alert, the userhas a lower engagement than if the user were inputting into theapplication providing the alert. In this case, the appropriate output isa “Level 2” haptic output. For example, a Level 2 output might be twomicrotaps in a row.

Skipping to the second to last row, the user may have the applicationissuing the alert active, but operating in the background, and may notbe touching the device at the time of the alert condition. As a result,it may take a greater haptic to get the user's attention in thiscircumstance. Thus, a “Level 3” haptic output may be appropriate, e.g.,a priming tap followed by a microtap or series of microtaps.

Finally, the last row shows a user with the application issuing thealert in an inactive state. In some circumstances, the application isnot executing even in the background, or the device is asleep. In thisinstance, the maximum haptic, “Level 4,” is appropriate to get theuser's attention. For example, a Buzz-Buzz haptic output might beappropriate in some circumstances. According to some embodiments, moreintense haptics (e.g., higher amplitude or longer haptics) are used forunexpected alerts that are not responsive to user input than thosehaptics used for feedback that is in direct response to user inputs.

This is just one example, using a limited number of circumstances andhaptic output examples, and assuming that all applications and alertsare treated the same, which may not be the case in other circumstances.In addition, a particular level of haptic output may be a “default”output, e.g., Level 2, and then depending on the user's focus the devicemay adjust up or down the intensity of the haptic output. Table 4provides conditions and corresponding outputs according to someembodiments, but is merely exemplary; other combinations of alerts,states, focuses, conditions and outputs are used according to differentembodiments.

In some embodiments, the haptic output is modified based on the locationof the user input. In this example, the device 100 detects a respectivealert condition and determines that the respective alert conditionresulted from receiving a tactile user input on the device. The device100 modulates a spatial haptic energy profile of the first hapticoutput, by reducing the amplitude of haptic energy of the first hapticoutput such that it is perceptible at the location of receiving thetactile user input on the device. The reduced amplitude of the hapticenergy may be such that it is not perceptible at locations farther fromthe user's finger/tactile input location, according to some embodiments.For example, the low amplitude may be perceptible by finger at the pointof contact, but not the rest of the hand and/or the wrist. Similarly,the device 100 in some embodiments detects a respective alert conditionand determines that the respective alert condition resulted fromreceiving a tactile user input on the device, but then specificallyidentifies a location of receiving the tactile user input on the deviceand modulates a spatial haptic energy profile of the first haptic outputby providing greater than a specified proportion of haptic energy of thefirst haptic output within a specified radius threshold of the locationof receiving the tactile user input on the device. According to someembodiments, the location of receiving the tactile user input on thedevice varies with time, and thus the spatial haptic energy also varieswith time corresponding to the time-varying location of receiving thetactile user input.

According to some embodiments, morphological characteristics of thehaptic output mimic the tactile input. For example, the device 100detects a respective alert condition and determines that the respectivealert condition resulted from receiving a tactile user input, and thenidentifies time-varying morphological attributes of the tactile userinput. Time-varying morphological attributes of the tactile inputinclude detection of a contact on a touch-sensitive surface, monitoringwhether the contact is maintained with the surface and how far and inwhich direction it moves, determining amount of pressure placed on thepoint of contact, etc., according to various embodiments. Then, thedevice 100 modifies, over time, morphological characteristics of thefirst haptic output to mimic the time-varying morphological attributesof the tactile user input. For example, morphological characteristics ofthe first haptic output to mimic include varying the amplitude,duration, frequency, etc. of the haptic output to mirror the tactileinput.

In some embodiments, waveforms of the haptic components need not beperfectly aligned with the user tactile input, and the device 100 cantake into account the fact that the haptic outputs will be perceived bya user to be concurrent or simultaneous with the tactile input even ifthey are slightly offset in time (e.g., because the haptic output isprocessed more quickly than the user can perceive it as a delayedresponse, thus causing a user to perceive the haptic output as occurringconcurrently or simultaneously). In some embodiments, specific hapticfeatures of the available predetermined set of haptic features areselected to constitute the haptic component of the first output, theselection based on which haptic features best mimic the user input. Someexamples include a microtap for a simple touch, pen-click tap for alonger/deeper touch, buzz for a swipe, buzz with a boost at the end fora swipe ending in finger lift-off. For example, the time-varyingmorphological attributes include a time-dependent contact pressureprofile between the tactile user input and a touch-sensitive surface ofthe computing device according to some embodiments, and the modifyingcomprises modulating, with time, an energy profile of the first hapticoutput to mirror the time-dependent contact pressure profile between thetactile user input and the touch-sensitive surface of the computingdevice.

According to some embodiments, the haptic output can mimic the timeduration of a sustained user input, gesture, mechanical adjustment, orother device manipulation by the user. For example, the device 100 firstdetects a respective alert condition, and determines that the respectivealert condition resulted from a sustained user input. The sustained userinput includes tactile user input on a touch-sensitive display on thecomputing device, rotting a knob, pushing a button, or any other userinput of device manipulation with a detectable start/end time, accordingto some embodiments. In some embodiments, the device can determine thatthe alert condition resulted from a gesture, mechanical adjustment, orthe like. Responsive to the determining, the device 100 initiates thefirst haptic output at an initiation time of the sustained user input,detects termination of the sustained user input, and then terminates thefirst haptic output at a termination time of the sustained user input(e.g., mimicking the user input). In some embodiments, the haptic outputis initiated by/at a start time of the user input and terminates at anend time of the user input. Therefore, the haptic feedback/outputextends substantially throughout the duration of time of the user input.Haptic output could be provided at location of user input, not limitedto a touch-sensitive surface. For example, if a rotating knob on theside of a watch is turned, a haptic that feels like a click couldprovide sustained feedback as the user turns the knob.

In some embodiments, prior to providing the second haptic output, thedevice 100 determines 1035 whether an ambient condition-based modulationcondition exists, and responsive to determining that an ambientcondition-based modulation condition exists, and prior to providing thesecond haptic output, the device modifies 1040 the second haptic output.

For example, the device 100 determines whether an ambient conditionexists as a potential hindrance to the user perception of the hapticoutput. In this example, the device 100 determines whether one or moreambient conditions exist for the device as potential interference tohaptic output at a time of occurrence of the respective alert condition.Ambient conditions include, according to some embodiments: noise (caninclude measured ambient noise as well as identifying that the user isin a library, museum, movie theatre, hospital or other place generallyknown to be quiet), movement/vibration impacting the user's physicalreceptiveness to a haptic sensation, engagement level with the device(e.g., as discussed above: viewing, touching, talking to), userproximity to the device (e.g., device on the user's person vs. locatedat greater than a specified proximity threshold from the device, etc.).

TABLE 5 AMBIENT CONDITIONS/HAPTIC FEEDBACK Primary Application AmbientCondition State Haptic Modification Noise (Low) Active None Noise (Low)Inactive None Noise (High) Active Increase 1 Noise (High) InactiveIncrease 2 Vibration (Low) Active None Vibration (Low) Inactive Increase1 Vibration (High) Active Increase 1 Vibration (High) Inactive Increase3; Delay Distance to device Active None (Short) Distance to deviceInactive None (Short) Distance to device Active Increase 1 (Long)Distance to device Inactive Increase 3; Delay (Long) Engagement levelActive Decrease 1 (High) Engagement level Active None (Low)

Table 5 shows various ambient conditions that can be present duringalert conditions, and for various application states, and possiblemodifications to the haptic output provided based on those conditionsaccording to some embodiments. The ambient conditions shown include lowand high noise levels, low and high vibration levels, short and longdistances to a device, and high and low user engagement levels, whichties into the above section and Table 4. The table is set up withexamples of various ambient conditions and for a given application state(active or inactive), how the haptic output might be modified to adjustfor the ambient condition. Table 5 only shows modification to hapticoutput, but similar adjustments could be made to non-haptic output, suchas audio. The Haptic Modification column lists possible modifications ofNone (maintain default haptic for the alert), Increase 1, 2, or 3, andDecrease 1, although many variations are possible; these are justexamples. For some conditions, such as High Noise, Low Vibration(Inactive), High Vibration (Active), and Long Distance to Device (rows3, 4, 6, 7, and 11), the device can modify the default haptic byincreasing it one or two steps, e.g., a microtap to a tap or twomicrotaps, or a tap to a buzz, etc., in these ambient conditions thatmay cause some level of distraction to the user. For other conditions,such as High Vibration (Inactive) or Long Distance to Device (rows 8 and12), the device can modify the default haptic by increasing it by agreater amount, e.g., a microtap to a buzz, etc., in these ambientconditions that may cause a greater level of distraction to the user. Insome circumstances, the haptic may be accompanied by audio output aspart of the modification. Alternatively or in addition to modifying theintensity of the haptic, the device may delay the haptic output untilthe ambient condition becomes more conducive to the user receiving thehaptic output, e.g., for the ambient conditions marked Increase 3. Inthis example, the device may periodically determine the ambientconditions to ascertain if the interfering ambient condition hassubsided and thus the haptic may be output. In other circumstances, nodelay is applied. These examples are very limited, using a limitednumber of circumstances and haptic output examples, and assumes that allapplications and alerts are treated the same, which may not be the casein other circumstances.

In some embodiments, no modification is required because the ambientcondition does not cause the user much if any distraction (e.g., rowswith “none” in the modification column). Finally, under some ambientconditions the device may reduce the default haptic associated with thealert condition, e.g., for a high engagement level, in which case lesseroutput is needed since the user is engaged. Table 5 provides conditionsand corresponding outputs according to some embodiments, but is merelyexemplary; other combinations of ambient conditions, alerts, states, andoutputs are used according to different embodiments.

According to some embodiments, the device 100 delays the haptic output.First the device 100 makes a determination that no interferentialambient conditions exist for the device. In this example, the device 100provides a respective second haptic output based on the state ofreceptiveness of the user, and in accordance with a determination thatone or more ambient conditions exist for the device as potentialinterference to haptic output, the device 100 delays providing therespective second output to the user. For example, a delay may continueuntil a subsequent instance in which no interferential ambientconditions exist (e.g., the device 100 determines again at some laterpoint in time whether the interfering ambient condition exists). Forexample, if the user is very active (e g, running or in the midst of aworkout) when the notification is received, and is therefore unable toperceive the haptic sensation, the device can delay the haptic outputuntil activity is completed, such as waiting until the user hascompleted the activity, and then provide the haptic output correspondingto the notification. Under other circumstances the alert is ofsufficient urgency that the device may not delay it regardless ofambient condition. In these examples, the determination of ambientconditions act as a proxy for estimating the user's physicalreceptiveness to a haptic sensation, for example, based on properties ofthe user's current surroundings.

According to some embodiments, the device 100 provides a variant of thehaptic output, e.g., a modification as described in conjunction withTable 5. In this example, the device 100 detects a respective alertcondition associated with a respective application and determines thatthe respective alert condition did not result from receiving user input.Responsive to determining that the respective alert condition did notresult from receiving user input, the device 100 determines whether oneor more ambient conditions exist for the device as potentialinterference to haptic output at a time of occurrence of the respectivealert condition, and in accordance with a determination that nointerferential ambient conditions exist for the device, the device 100provides a first variant of the second haptic output to the user, thefirst variant of the second haptic output having a first set of outputcharacteristics. For example, the variant can be a haptic output that issofter, of shorter duration, includes more subtle haptic features (e.g.,microtaps), according to some embodiments. In accordance with adetermination that one or more ambient conditions exist for the deviceas potential interference to haptic output, the device 100 provides asecond variant of the second haptic output to the user, the secondvariant of the second haptic output having a second set of outputcharacteristics, the second set of characteristics different from thefirst set of characteristics, and the second variant of the secondhaptic output having greater intensity than the first variant of thesecond haptic output.

In some embodiments, the device 100 can modify the haptic output to“prime” the user. For example, a device 100 first detects a respectivealert condition associated with a respective application and thendetermines that the respective alert condition did not result fromreceiving user input. Responsive to determining that the respectivealert condition did not result from receiving user input, the deviceprovides a priming haptic output as a precursor to the second hapticoutput, the priming haptic output designed to increase an engagementlevel of a user operating the device to the second haptic output,wherein at least a portion of the priming haptic output is morepronounced than the second haptic output, according to some embodiments.The device 100 optionally provides the second haptic output followingand within a specified time interval of providing the priming hapticoutput.

The device 100 can provide variations of priming according to someembodiments. In one example, the priming haptic output comprises apriming haptic component with a time-varying waveform morphologycharacterized by a progressively increasing intensity of the hapticcomponent with time. In some embodiments, the priming haptic outputincludes a gradually increasing intensity of haptic waveform tosubconsciously prepare user for notification. After priming, a largeincrease in audio volume occurs in concert with the haptic componentaccording to some embodiments. In another example the priming hapticoutput optionally comprises a priming haptic component with atime-varying waveform morphology characterized by an emphasized hapticfeature followed by a progressively diminishing intensity of the hapticcomponent. Alternatively, the device 100 can deliberately prime theuser's sensitivity by providing a stronger priming haptic (andoptionally audio) output first and then follow it up with milder outputsonce the user's senses are primed, according to some embodiments.According to some embodiments, the second haptic output is preceded bythe priming haptic output and the first haptic output is provideddirectly, without a corresponding preceding priming haptic output, inresponse to a corresponding alert condition that results from a receiveduser input. In various embodiments, characteristics of the priminghaptic output are selected based on an urgency or context associatedwith the alert condition, with the intensity of the output mirroring theurgency of the alert.

It should be understood that the particular order in which theoperations in FIG. 10 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) are alsoapplicable in an analogous manner to method 1000 described above withrespect to FIG. 10. For example, the inputs, alert conditions,applications, and haptic outputs described above with reference tomethod 1000 optionally have one or more of the characteristics of theinputs, alert conditions, applications, and haptic outputs describedherein with reference to other methods described herein (e.g., methods700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200). For brevity,these details are not repeated here.

Haptic Output for Multi-Part Operations

Whether a detected alert condition is part of a multi-part operationprovides the basis for providing a corresponding haptic output,according to some embodiments. Providing haptic output that correlateswith whether an alert condition is part of a multi-part operationcreates a more efficient human-machine interface, thereby reducing theamount of time it takes for a user to perform operations, whichconsequently reduces energy use and increases battery life for batterypowered devices. FIG. 11 is a flow diagram of a method 1100 fordetecting an alert condition associated with a multi-part operation andproviding a corresponding haptic output according to some embodiments.Note that in some embodiments, different steps may be performed otherthan those shown in FIG. 11.

The method begins by receiving 1105 an input that corresponds to a firstpart of a multi-part operation performed by an application executing ona computing device 100. In some embodiments, the input corresponding tothe first part of the multi-part operation is detected on atouch-sensitive surface of the device 100.

Responsive to receiving the input corresponding to the first part of themulti-part operation, the device 100 initiates 1110 an ongoing hapticoutput sequence. The outgoing haptic sequence can be ongoing in thesense that it is sustained, constant, varying but continuing over aperiod of time, or intermittent according to various embodiments, solong as it continues until the second input is received. In someembodiments, responsive to detecting the input corresponding to thefirst part of the multi-part operation, the device also initiates anongoing audio output sequence to accompany the ongoing haptic output.

According to some embodiments, the device 100 has a touch-sensitivesurface, and the haptic output sequence is provided via thetouch-sensitive surface. For example, the haptic output includes hapticvibration of the touch-sensitive surface in a predetermined pattern thatis perceptible to a user of the device that is touching thetouch-sensitive surface.

In some embodiments, in addition to the ongoing haptic output sequence,the device 100 provides 1115 a first haptic feedback responsive toreceiving the input corresponding to the first part of the multi-partoperation. The first haptic feedback is selected based on the event andindicating an initiation of the event, and is distinct from the ongoinghaptic output sequence in some circumstances. For example, the firsthaptic feedback has a greater intensity haptic output than the ongoinghaptic output sequence according to some embodiments.

After initiating the ongoing haptic output sequence, the device 100receives 1120 an input that corresponds to a second part of themulti-part operation according to some embodiments. For a device 100having a touch-sensitive surface, the input corresponding to the secondpart of the multi-part operation is detected on the touch-sensitivesurface.

In some embodiments two devices are involved. Thus, in some embodimentsthe inputs corresponding to both the first part of the multi-partoperation and the second part of the multi-part operation are detectedby the device 100. However, in some embodiments the input correspondingto the first part of the multi-part operation is detected by the device100 and the input corresponding to the second part of the multi-partoperation detected by a second device.

According to some embodiments, additional user input can be received bythe device 100. For example, after initiating 1110 the ongoing hapticoutput sequence responsive to the first part of the multi-partoperation, and before receiving 1120 the input that corresponds to thesecond part of the multi-part operation, the device 100 receives 1125one or more additional user inputs, distinct from the input thatcorresponds to the second part of the multi-part operation according tosome embodiments. In these circumstances, the device continues toprovide the ongoing haptic output regardless of the additional userinputs, until receipt of the input corresponding to the second part ofthe multi-part operation, according to some embodiments.

Responsive to receiving the input corresponding to the second part ofthe multi-part operation, the device 100 terminates 1130 the ongoinghaptic output sequence according to some embodiments. In someembodiments, responsive to detecting the input corresponding tothesecond part of the multi-part operation, the device 100 alsoterminates any ongoing audio output sequence.

The input corresponding to the second part of the multi-part operationfollows the input corresponding to the first part of the multi-partoperation and is separated in time from the input corresponding to thefirst part of the multi-part operation by a specified interval of timecorresponding to a duration of the event according to some embodiments.Further, the ongoing haptic output sequence is provided throughout theduration of the event and indicates occurrence of the event in somecircumstances.

According to some embodiments, the device also provides 1135 a secondhaptic feedback responsive to receiving the input corresponding to thesecond part of the multi-part operation, the second haptic feedbackdistinct from the ongoing haptic output sequence. The second hapticfeedback indicates termination of the event, and is distinct from theongoing haptic output sequence in some circumstances. For example, thesecond haptic feedback has a greater intensity haptic output than theongoing haptic output sequence according to some embodiments.

The multi-part operation can take various forms according to differentembodiments. According to some embodiments the multi-part operation is asecure transaction, with the first part of the multi-part operationincluding arming the device to authorize the secure transaction. In thisexample, in response to the user providing credentials, e.g., a passwordor biometric authentication such as a fingerprint, the device is putinto in a payment mode where payment information can be provided to apayment terminal via a near field communication (NFC) subsystem. Thus,the arming of the device is the first part, and would initiate acorresponding ongoing haptic output. In addition, the second part themulti-part operation includes authorizing the secure transactionaccording to this example. The authorizing thus is the second step andincludes, e.g., providing payment information to the payment terminalwhen the NFC subsystem comes into range of an NFC field of the paymentterminal while the NFC subsystem is armed from the first step accordingto some embodiments.

In some embodiments of a NFC multi-part secure transaction, the firstpart of the multi-part operation includes determining that userauthentication is required to complete the secure transaction. Forexample, in response to an NFC subsystem coming into range of an NFCfield of a payment terminal, prompting the user to provide credentialssuch as a password or biometric authentications such as a fingerprint toauthorize the secure transaction. In this example, the second part themulti-part operation includes receiving user authentication for thesecure transaction. For example, the haptic output is provided to reminda user that the user needs to provide user authentication informationbefore the payment process can proceed, the user authenticationinformation optionally includes passcode based or biometric basedauthentication. In some embodiments, a first haptic output pattern isprovided by the device between when the need for user authentication isdetermined and when user authentication is received (e.g., betweenrequesting user authentication and receiving user authentication) toremind the user to provide user authentication information, and a secondhaptic output pattern is provided by the device between when the userauthentication information is received and the secure transaction isauthorized (e.g., between receiving user authentication and providingthe payment information to the payment terminal). In this example, thesecond haptic output pattern is different from the first haptic outputpattern to remind the user to place the NFC subsystem near the paymentterminal according to some embodiments.

According to other payment embodiments, the input corresponds to asecure transaction that does not use NFC. In this example, the firstpart of the multi-part operation is a selection of a first userinterface element that initiates a transaction. For a financialtransaction such as an online purchase, selection of on elementcorresponding to activating an online payment enables a user to enter auser interface for entering secure or confidential information. Asanother example, a user requests to log into a secure location, such asecure email account or online retailer. Continuing the above example,the multi-part operation comprises one or more inputs for entry ofinformation required to process the transaction. For example, entry ofcredit card information and other verification information necessary toauthorize the financial transaction, and a corresponding ongoing hapticoutput sequence is provided throughout the duration of entry of theinformation. Similarly the input may be entry of username and passwordto log into a secure web account. In some embodiments, the eventcorresponds to population of one or more data entry fields that receivethe information required to process the transaction and the inputcorresponding to the second part of the multi-part operation is aselection of a user interface element that authorizes completion of thetransaction, e.g., an authorize payment user interface element.

According to some embodiments, the first part of the multi-partoperation includes closing a draft of a document, and the second part ofthe multi-part operation includes returning to the draft of thedocument. Examples of closing of the document include closing a draft ofan email or text message to switch to another task or to open anotherapplication on the device and returning to the document includesswitching back to the draft of the email or text message from the othertask or other application, according to various embodiments.

In some embodiments the inputs are separate inputs occurring closetogether in time. For example, the input corresponding to the first partof the multi-part operation is a first user input interacting with anapplication executing on the device and the input corresponding to thesecond part of the multi-part operation is a second user inputinteracting with the application, with the second user input distinctfrom the first user input according to some embodiments. For instance,the first user input into the application could be a first tap andliftoff, and the second user input could be a second tap and liftoffwithin the same application.

In some circumstances the multi-part operation is a multi-part gesture.For example, the operation comprises first detecting the input thatcorresponds to the first part of the multi-part operation as a contacton a touch-sensitive surface of the device 100 and detecting the inputthat corresponds to the second part of the multi-part operation includesdetecting movement of the contact on the touch-sensitive surfaceaccording to some embodiments. In the example of a swipe gesture, thefirst part is a user touching the touch-sensitive surface, and thesecond part is the user maintain in the contact with the surface whilemoving the contact across the touch surface. In another example, theoperation comprises first detecting the input that corresponds to thefirst part of the multi-part operation as movement of a contact on thetouch-sensitive surface and detecting the input that corresponds to thesecond part of the multi-part operation as detecting liftoff of thecontact from the touch-sensitive surface.

According to some embodiments, the inputs corresponding to the first andsecond parts of the multi-part operation comprise a single gesture,wherein the input corresponding to the first part of the multi-partoperation is an initial part of the gesture (e.g., touch and hold with afinger contact) and the input corresponding to the second part of themulti-part operation is a subsequent part of the gesture (e.g., aliftoff of the finger contact). For example, in a text editing orviewing application, holding a finger down on a word first highlightsthe word and then continuing to hold the finger down on the highlightedword invokes a cut-copy-paste menu. In this example, the liftoff of thecontact would represent selection of a displayed aspect corresponding tothe touch and hold.

In some embodiments, the gesture is initiated on a portion of thetouch-sensitive surface that corresponds to a first location in a userinterface displayed on a display of the device 100 and terminates on asecond portion of the touch-sensitive surface that corresponds to asecond location in the user interface that is different from the firstlocation. For example, a touch-swipe-liftoff sequence can be used toperform a drag and drop, or to relocate a focal region in a cameraviewer, according to some embodiments.

According to some embodiments, the input corresponding to the first partof the multi-part operation initiates an event associated with therespective application and the event is sustained until receipt of theinput corresponding to the second part of the multi-part operation, atwhich time the input corresponding to the second part of the multi-partoperation terminates the event associated with the respectiveapplication. For example, the first input is a selection of a userinterface element (e.g., provided on the application user interface ofan application) that initiates an event associated with the applicationand the second input is a selection of a different user interfaceelement (e.g., on the application user interface of the sameapplication) that terminates the event associated with that applicationaccording to some embodiments.

The application is a text editing application according to someembodiments, and the input corresponding to the first part of themulti-part operation is a selection of a first user interface elementthat enables a text-entry mode of the text editing application. In thisexample, the multi-part operation comprises one or more inputs fortext-entry to the text editing application, the event corresponds totext manipulation in the text editing document responsive to the inputsfor text-entry, and the input corresponding to the second part of themulti-part operation is a selection of a user interface element thatdisables the text-entry mode of the text editing application. Accordingto some embodiments, the text editing application is a communicationapplication such as an email or messaging application that allows thedevice user to communicate with other users. According to someembodiments, the text editing application is a non-communication textediting application, such as a word processing application, a notepad orsticky note application, or other application that allows text entry butdoes not provide independent communication functionality.

According to some embodiments, characteristics of the ongoing hapticoutput sequence are selected based on the event that is initiated by theinput corresponding to the first part of the multi-part operation. Forexample, putting a call on hold is accompanied by a lower intensity(e.g., lower energy, less invasive, less perceptible) haptic outputsequence than a financial transaction event or a secure login, for whichthe user has provided confidential or secure information and thus is atransaction that should garner additional attention. In this example,throughout the duration of the multi-part operation (e.g., after theconfidential information is entered but before the transaction iscomplete) the user is thus reminded of the prior entry such that he orshe completes the transaction quickly so as to avoid compromising theconfidentiality of the information being entered. In some circumstances,a timeout function is used so that the confidential information is notleft available. Additionally, different haptic output patternsoptionally provide the user with information as to what task they are inthe middle of completing.

The input corresponding to the first part of the multi-part operationsuspends an ongoing event associated with the application, and the inputcorresponding to the second part of the multi-part operation resumes theongoing event associated with the application according to someembodiments. For example, the first part initiates a pause in an ongoingevent in some circumstances. In some embodiments, the first inputcorresponds to a selection of a user interface element that pauses anaction, such as playing of a multimedia file, a video clip, a song, or aphone call in progress, and the second input is a selection of adifferent) user interface element that resumes the action.

According to some embodiments, an intensity of the ongoing haptic outputsequence increases with time until a subsequent input corresponding tothe multi-part operation is received. For example, haptic energy (e.g.,amplitude, frequency, haptic features) of the ongoing haptic outputsequence is modulated (e.g., via increasing intensity or haptic energy)over time to remind the user with increasing urgency to provide furtherinput corresponding to the multi-part operation. Such an increasingoutput can correspond to a reminder to terminate the event that wasstarted by the first part of the multi-part input, or to resume theevent that was suspended by the first part of the multi-part inputaccording to various embodiments.

In some embodiments, a time-varying haptic energy profile of the hapticoutput sequence mimics a time-varying acoustic energy profile of theaudio output sequence. As discussed in conjunction with FIG. 6, thetime-varying waveform morphologies of the haptic and audio sequencesmimic, mirror, align, and/or synchronize with each other according tosome embodiments.

According to some embodiments, the output includes audio outputdepending on whether the first part of the multi-part operationinitiates a new event or pauses an ongoing event. Responsive todetermining that the input corresponding to the first part of themulti-part operation does not pause an ongoing event, but initiates anevent associated with the respective application, the device 100provides the haptic output sequence without an accompanying audio outputsequence, and responsive to determining that the input corresponding tothe first part of the multi-part operation pauses an ongoing eventassociated with the respective application, the device 100 provides anongoing audio output sequence accompanying the ongoing haptic outputsequence according to some embodiments. In some embodiments, the audiooutput sequence accompanies the haptic output sequence only when thefirst part of the multi-part operation pauses or suspends an ongoingevent or action. During the pause, the user is presumably not as engagedwith the device and/or application. For instance, the user may bepassively waiting for a call to be transferred, and therefore cantolerate additional stimulus associated with an audio output. However,in some embodiments, the audio sequence does not accompany the hapticsequence if an actual event is initiated by the first part of themulti-part operation according to some embodiments, such as a useractively typing an email message or entering secure financial or log ininformation.

The application is a voice communication application according to someembodiments. For example, the application can be a regular phoneapplication, a voice over internet application such as Skype, or a voicecommunication application embedded in an email application such as Gchataccording to some embodiments. The event is an ongoing voicecommunication occurring at a time of receipt of the input correspondingto the first part of the multi-part operation and the inputcorresponding to the first part of the multi-part operation is aselection of a first user interface element of the voice communicationapplication, the selection suspending the ongoing voice communicationaccording to some embodiments. For example, a voice call is paused dueto a call transfer or call on hold request. The input corresponding tothe second part of the multi-part operation is a selection of a seconduser interface element that resumes the voice communication according tosome embodiments.

It should be understood that the particular order in which theoperations in FIG. 11 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1200, 1300, 2500, 2600, 3000, and 3200) are alsoapplicable in an analogous manner to method 1100 described above withrespect to FIG. 11. For example, the inputs, operations, applications,and haptic outputs described above with reference to method 1100optionally have one or more of the characteristics of the inputs,operations, applications, and haptic outputs described herein withreference to other methods described herein (e.g., methods 700, 900,1000, 1200, 1300, 2500, 2600, 3000, and 3200). For brevity, thesedetails are not repeated here.

Haptic Output for Subset Operations

Whether a detected request to perform an operation related to anoperation that is a subset of another operation provides the basis forproviding a corresponding haptic output, according to some embodiments.Providing haptic output that correlates with whether a request toperform an operation pertains to an operation that is a subset ofanother operation creates a more efficient human-machine interface,thereby reducing the amount of time it takes for a user to performoperations, which consequently reduces energy use and increases batterylife for battery powered devices. FIG. 12 is an exemplary method flowdiagram of a method 1200 for detecting first and second inputs toperform first and second operations, performing the operations, andproviding corresponding outputs according to some embodiments. Note thatin some embodiments, different steps may be performed other than thoseshown in FIG. 12.

The method begins by detecting 1205 a first input that corresponds to arequest to perform a first operation. According to some embodiments, thecomputing device 100 comprises a touch-sensitive display, and the firstinput is received through the touch-sensitive display according to someembodiments.

According to some embodiments, responsive to detecting the first input,the device 100 provides 1210 a first output that includes a hapticcomponent. If the comprises a touch-sensitive display, the hapticcomponent of the first output is provided via the touch-sensitivedisplay according to some embodiments.

Also responsive to detecting the first input according to someembodiments, the device 100 performs 1215 the first operation. The firstoperation can be any operation performed by the device 100 in responseto an input. The examples described below, according to variousembodiments, include capture of an image with a camera, atransaction-enable operation for enabling the device to authorize asecure transaction, a save operation for saving content within anexisting file, and a send operation for sending a reply to a messagethat is in a message inbox, but this list is not meant to be exhaustive;other operations by the device 100 are envisioned as following the stepsof the method.

After performing the first operation, the device 100 detects 1220 asecond input that corresponds to a request to perform a second operationthat includes the first operation and a supplemental operation accordingto some embodiments. If the computing device 100 comprises atouch-sensitive display, and the second input is received through thetouch-sensitive display, according to some embodiments.

According to some embodiments, responsive to detecting the second inputthe device 100 provides 1225 a second output that includes a hapticcomponent, wherein the second output includes the first output inconjunction with providing supplemental output that corresponds to thesupplemental operation. If the comprises a touch-sensitive display, thehaptic component of the second output is provided via thetouch-sensitive display according to some embodiments.

In some embodiments, the haptic component corresponding to the secondoutput comprises a first haptic component corresponding to the firstoutput and a second haptic component corresponding to the supplementaloutput, and the second haptic component is distinct from the firsthaptic component. For example the second haptic component has a higherintensity (e.g., amplitude, duration, frequency) than the first hapticcomponent according to some embodiments.

The supplemental output comprises a non-haptic component according tosome embodiments. The non-haptic component can be any of audio or visual(e.g., graphic, textual) according to various embodiments. In someembodiments the supplemental output comprises an audio component but thefirst output does not include an audio component. For example, an audiooutput is provided during an interval when a camera application measuresthe specified interval of time counting down to the image capture, butthe operation of capturing the image itself is not accompanied by anaudio sound.

For supplemental output including audio output, a haptic output isaccompanied by a corresponding audio output according to someembodiments, and the audio and haptic portion are aligned when at leasta portion of the audio output is concurrent with at least a portion ofthe haptic output, or the audio output occurs close in time to thehaptic output according to some embodiments. The haptic and audiowaveforms need not be perfectly aligned, and the device 100 can takeinto account the fact that for certain classes of haptic and audiooutputs, the haptic and audio outputs will be perceived by a user to beconcurrent or simultaneous even if they are slightly offset in time(e.g., because audio output is processed more quickly than hapticoutput, so providing haptic output before providing audio output will,in some circumstance, cause a user to perceive the audio and hapticoutput as occurring concurrently or simultaneously). Variations in theaccompanying audio portion also can vary between embodiments thatinclude an audio output. For example, each of an amplitude of an audiocomponent associated with the first output, a duration of the audiocomponent associated with the first output, one or more musicalqualities (e.g., pitch, tone, etc.) associated with the audio componentassociated with the first output, can vary in certain circumstances.

In some embodiments, the same haptic waveform generates differentperceptions when accompanied by different audio waveforms. For example,if an audio (ring) tone is played with a high pitch versus a low pitch,and accompanied by the same haptic component (or substantially similarhaptic components that a typical user will not be able to differentiatebetween), the perception of the haptic component differs despite theidentical haptic. The higher pitched or shrill version of the ring tonemight be used with the second output to draw elevated user attention. Insome embodiments, the first haptic output is similar, identical, orsubstantially identical to the second haptic output.

According to some embodiments, the second output comprises a textualcomponent that identifies the supplemental operation as different fromthe first operation. For example, the textual component may describe thesupplemental information. For instance, an output corresponding to asend and archive operation provides a textual information indicatingthat the message has been archived. As another example, an outputcorresponding to a payment success message indicates that the secureinformation has been validated.

The supplemental operation has a variable value and the supplementaloutput is based on the value of the variable attribute according to someembodiments. For example, a time interval before a picture is taken by acamera can be five seconds, 10 seconds, 15 second, etc. according tovarious embodiments. For a 10 second camera timer, the supplementaloutput thus correspondingly lasts for 10 seconds according to someembodiments.

According to some embodiments, the device 100 then performs 1230 thesecond operation. The timing of the performance of the first and secondoperations, respectively, can vary. According to some embodiments, thefirst operation is performed concurrently with providing the firstoutput and the supplemental operation is performed concurrently withproviding the supplemental output. For example, for a shutterself-timer, the first output is provided when the timer counts down,which corresponds to the first operation, and the supplemental output isprovided when the shutter takes the picture, corresponding to thesupplemental operation.

The first operation can be performed prior to performing thesupplemental operation and the first output provided prior to providingthe supplemental output according to some embodiments. For example, fora first operation corresponding to a send message (e.g., email)function, a second operation is a send and archive operation, and thesend operation is performed prior to the supplemental operation, whichis the archiving portion.

According to some embodiments, at least a portion of the first operationis performed concurrently with a portion of the supplemental operationand at least a portion of the first output is provided concurrently witha portion of the supplemental output.

A few examples follow that help to illustrate the steps of the method1200. In a first example, the first operation corresponds to capture ofan image with a camera. The camera is integrated into the device 100 oris remote from the device but is remotely controlled by the device,according to various embodiments. The second operation in this examplecorresponds to capturing an image after a specified interval of time.Specifically, the method of this example begins by detecting a firstinput that corresponds to a request to capture of a first image with acamera, and responsive to detecting the first input, provides a firstoutput that includes a haptic component according to some embodiments.Also responsive to detecting the first input, the device 100 capturesthe first image. After capturing the first image, the device 100 detectsa second input that corresponds to a request to capture a second imageafter a specified interval of time, and responsive to detecting thesecond input provides a second output that includes a haptic componentaccording to some embodiments. In this example, the second outputincludes the first output in conjunction with providing supplementaloutput that corresponds to measuring lapse of the specified interval oftime and the second image is captured after the specified interval oftime.

In another example, the first operation corresponds to atransaction-enable operation for enabling the device to authorize asecure transaction and the second operation corresponds to enabling thedevice to authorize a secure transaction and authorizing the securetransaction according to some embodiments. For example, in an onlinepayment context in which credit card information is needed, the enableoperation prepares an NFC subsystem to provide payment credentials at apayment terminal or other point of sale, or a user log in request thatneeds user credentials and password. Specifically, according to thisexample the method begins by detecting a first input that corresponds toa request to perform a transaction-enable operation for enabling thedevice to authorize a secure transaction, and responsive to detectingthe first input, providing a first output that includes a hapticcomponent. Also responsive to detecting the first input, enabling thedevice to authorize a secure transaction. After performing thetransaction-enable operation, detecting a second input that correspondsto a request to authorize a secure transaction, and responsive todetecting the second input, providing a second output that includes ahaptic component, wherein the second output includes the first output inconjunction with providing supplemental output that corresponds toauthorizing the secure transaction the supplemental operation. Alsoresponsive to detecting the second input, authorizing the securetransaction.

The first operation corresponds to a save operation for saving contentwithin an existing file, and the second operation corresponds to asave-as operation to save content within an existing file to a new fileaccording to some embodiments. In this example, a first input isdetected that corresponds to a request to perform a save operation, andresponsive to detecting the first input, a first output is provided thatincludes a haptic component. Also responsive to detecting the firstinput, the save operation is performed for saving content within anexisting file. After performing the save operation, a second input isdetected that corresponds to a request to perform a save as operation tosave content within an existing file to a new file, and responsive todetecting the second input, a second output is provided that includes ahaptic component, wherein the second output includes the first output inconjunction with providing supplemental output that corresponds to acreation of the new file according to some embodiments. Also responsiveto detecting the second input, the save as operation is performed.

According to some embodiments the first operation corresponds to a sendoperation for sending a reply to a message that is in a message inboxand the second operation corresponds to a send and archive operation forsending a reply to a message that is in the message inbox and removingit from the message inbox.

It should be understood that the particular order in which theoperations in FIG. 12 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200) are alsoapplicable in an analogous manner to method 1200 described above withrespect to FIG. 12. For example, the inputs, operations, applications,components, and haptic outputs described above with reference to method1200 optionally have one or more of the characteristics of the inputs,operations, applications, components, and haptic outputs describedherein with reference to other methods described herein (e.g., methods700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200). For brevity,these details are not repeated here.

Alert Condition Class-Associated Haptic Output

Whether two detected alert conditions are in the same or different classof alert condition provides the basis for providing a correspondinghaptic output, according to some embodiments. Providing haptic outputthat correlates with whether alert conditions are in the same ordifferent class of alert condition creates a more efficienthuman-machine interface, thereby reducing the amount of time it takesfor a user to perform operations, which consequently reduces energy useand increases battery life for battery powered devices. FIG. 13 is anexemplary method flow diagram of a method 1300 for detecting first andsecond alert conditions and providing corresponding outputs according tosome embodiments. Note that in some embodiments, different steps may beperformed other than those shown in FIG. 13.

The method begins by detecting 1305 occurrence of a first alertcondition according to some embodiments. The first alert condition canbe any of the various alert conditions described elsewhere herein.

According to some embodiments, a first output is provided 1310 thatincludes a first haptic component and a first non-haptic componentresponsive to detecting the occurrence of the first alert condition. Thenon-haptic component can be any of audio or visual (e.g., graphic,textual) according to various embodiments. When an audio component isincluded, the audio and haptic components are aligned according to someembodiments. The haptic and audio waveforms need not be perfectlyaligned, and the device 100 can take into account the fact that forcertain classes of haptic and audio outputs, the haptic and audiooutputs will be perceived by a user to be concurrent or simultaneouseven if they are slightly offset in time (e.g., because audio output isprocessed more quickly than haptic output, so providing haptic outputbefore providing audio output will, in some circumstance, cause a userto perceive the audio and haptic output as occurring concurrently orsimultaneously). According to some embodiments, the non-haptic portionincludes a textual component.

After providing the first output, the device 100 detects 1315 occurrenceof a second alert condition, and responsive to detecting the occurrenceof the second alert condition, provides 1320 a second output thatincludes a second haptic component and a second non-haptic componentaccording to some embodiments. When the first alert condition and thesecond alert condition are different alert conditions that are in a sameclass of alert condition, the first output and the second output shareone or more same components and have one or more different componentsaccording to some embodiments. Alternatively, when the first alertcondition and the second alert condition are in a same class of alertcondition, the first and second outputs have one or more components incommon. When the first alert condition and the second alert conditionare different alert conditions that are in different classes of alertcondition, the first haptic component is different from the secondhaptic component and the first non-haptic component is different fromthe second non-haptic component according to some embodiments.Alternatively, when the first alert condition and the second alertcondition are in different classes of alert condition, the first andsecond outputs have no common components.

Classes of alert conditions include messages, activity alerts, mediaplayback alerts, virtual assistant alerts, system alerts, schedulingreminders, and internet browser updates according to some embodiments.

According to some embodiments, when the first alert condition and thesecond alert condition correspond to separate instances of the samealert condition, the first output is identical to the second output. Ifthe computing device 100 has a touch-sensitive display, the first hapticcomponent of the first output and the second haptic component of thesecond output are provided via the touch-sensitive display on thecomputing device.

In some embodiments, the first and second non-haptic components areaudio outputs. For non-haptic output including audio output, a hapticoutput is accompanied by a corresponding audio output according to someembodiments, and the audio and haptic portion are aligned when at leasta portion of the audio output is concurrent with at least a portion ofthe haptic output, or the audio output occurs close in time to thehaptic output according to some embodiments. The haptic and audiowaveforms need not be perfectly aligned, and the device 100 can takeinto account the fact that for certain classes of haptic and audiooutputs, the haptic and audio outputs will be perceived by a user to beconcurrent or simultaneous even if they are slightly offset in time(e.g., because audio output is processed more quickly than hapticoutput, so providing haptic output before providing audio output will,in some circumstance, cause a user to perceive the audio and hapticoutput as occurring concurrently or simultaneously). Variations in theaccompanying audio portion also can vary between embodiments thatinclude an audio output. For example, each of an amplitude of an audiocomponent associated with the first output, a duration of the audiocomponent associated with the first output, one or more musicalqualities (e.g., pitch, tone, etc.) associated with the audio componentassociated with the first output, can vary in certain circumstances.

In some embodiments, the same haptic waveform generates differentperceptions when accompanied by different audio waveforms. For example,if an audio (ring) tone is played with a high pitch versus a low pitch,and accompanied by the same haptic component (or substantially similarhaptic components that a typical user will not be able to differentiatebetween), the perception of the haptic component differs despite theidentical haptic. The higher pitched or shrill version of the ring tonemight be used with the second output to draw elevated user attention. Insome embodiments, the first haptic output is similar, identical, orsubstantially identical to the second haptic output.

In some embodiments, the first and second non-haptic components arevisual outputs, such as text, graphic or metadata output.

According to some embodiments, the first alert condition and the secondalert condition are in the same class of alert condition, wherein theclass is associated with alert conditions for an application. In thisexample, all alerts from a respective application fall within a commonclass, and classes are differentiated by application. For example, allvirtual assistant events optionally have same haptic component withdifferent audio components. In another example, incoming and outgoingmessages have the same haptic output with different audio, the samehaptic output optionally based on being associated with a common instantmessaging application. The common component (in this example, the hapticcomponent) identifies the application. In yet another example, eventsfor a personal automated assistant including confirm, cancel, and invokeevents optionally all have the same audio, but differing haptic outputs.

TABLE 6 MESSAGES Primary Alert Application Haptic Audio Condition StateMode Condition Output Output Message Active Messages No new X X receivedmessage Message Active Message No new X X received message MessageActive Draft No new X X received message Message Active Background Nonew X X received message Message Active Messages New Tap Messagereceived message Receive Default Message Active Message New Tap Messagereceived message Receive Default Message Active Draft New Tap Messagereceived message Receive Default Message Active Draft No new X X sentmessage Message Active Draft New Tap Message sent message Sent Default

Table 6 shows message received and message sent alert conditions for amessaging application according to some embodiments. In this example,various modes are listed for an active application state. In the case ofa new message received, as shown in rows five, six, and seven, atap-type haptic is output and a message received default audio isoutput. In the case of a new message sent, shown in the last row, atap-type haptic is output, and a message send default audio is output.Because message received (a first alert condition) and message sent (thesecond alert condition) are in the same class of alert condition inwhich the class is associated with an application (messaging), thehaptic output for both alert conditions is the same, but the audiooutput is different. Table 6 provides conditions and correspondingoutputs according to some embodiments, but is merely exemplary; othercombinations of alerts, states, modes, conditions and outputs are usedaccording to different embodiments.

According to some embodiments, the first alert condition and the secondalert condition are in the same class of alert condition, and the classis associated with a category of applications. For example, classes ofapplication include messaging applications, game applications,health/fitness applications, productivity applications, or socialnetworking applications according to some embodiments. In someembodiments, applications that have or perform common or similarfunctions optionally belong to a common class of applications. Examplesof classes of applications include, text-based communicationapplications (instant messaging, email, social media), voice-basedcommunication applications (phone, voice over internet), multimediaapplications (photo viewer applications, camera application, musicplayer, video player), text editing applications (word, notepad),scheduling applications (calendar, alarm), and internet browserapplications (chrome, safari). Classes are therefore differentiated byfunction across applications according to some embodiments. As oneexample, email messages and text messages although belonging todifferent applications (email and instant messaging) have the samehaptic output and different audio outputs according to some embodiments,by virtue of being associated with applications that are in the sameclass of applications (both text-based communication applications).

According to some embodiments, the first alert condition and the secondalert condition are in the same class of alert condition, and the classcorresponds to functions in a predefined class of functions. Forexample, alerts that report on accomplishment of activity goals withinan activity monitoring application (e.g., goal close, behind on goal,goal attained; achieved 10% of target, 50% of target, etc.) optionallyfall within the same class of alerts. Various alarms or reminders withina same scheduling application can be set for different predefined timeinstances, or to report on the satisfaction of various differentpredefined criteria according to some embodiments.

TABLE 7 ACTIVITY GOALS Primary Alert Application Haptic Audio ConditionState Mode Condition Output Output Sedentary Active Background No eventX X timer end Sedentary Active In use No event X X timer end SedentaryInactive N/A No event X X timer end Sedentary Active Background Timerend Tap Default timer end Activity Chime Sedentary Active In use Timerend Tap Default timer end Activity Chime Sedentary Inactive N/A Timerend Tap Default timer end Activity Chime Activity Active Background Noevent X X 50% of target Activity Active In use No event X X 50% oftarget Activity Inactive N/A No event X X 50% of target Activity ActiveBackground 50% goal Tap X 50% of attained target Activity Active In use50% goal Tap X 50% of attained target Activity Inactive N/A 50% goal TapX 50% of attained target Activity Active Background No event X X targetreached Activity Active In use No event X X target reached ActivityInactive N/A No event X X target reached Activity Active Background GoalTap Intense target attained Activity reached Chime Activity Active Inuse Goal Tap Intense target attained Activity reached Chime ActivityInactive N/A Goal Tap Intense target attained Activity reached Chime

Table 7 depicts various activity related goals according to someembodiments. In this example, three different alert conditions belongingto the same alert condition class are shown: sedentary end timer,activity 50% of target, and activity target reached. The class of alertconditions shown corresponds to functions in a predefined class offunctions of alerts that report on accomplishment of activity goalswithin an activity monitoring application. For each event in the table,the haptic output is a tap-type output, although the audio output foreach different type varies (e.g., default activity chime, no audio,intense activity chime). Table 7 provides conditions and correspondingoutputs according to some embodiments, but is merely exemplary; othercombinations of alerts, states, modes, conditions and outputs are usedaccording to different embodiments.

According to some embodiments, the first alert condition and the secondalert condition are associated with two different applications, whereinthe first alert condition and the second alert condition are in the sameclass of alert condition, the class corresponding to functions in apredefined class of functions within the two different applications.Examples of predefined classes of functions include, alerts that reporton incoming text communication from another user across applications(incoming email, incoming instant message, incoming social mediamessage) fall into the same class, alerts that report on incoming voicecommunication from another user across all applications (phone, voiceover internet, voice call within an email application) fall into acommon class, alerts that report application software upgrades for allapplications fall into a common class, system alerts that report onvarious states of the device (Wi-Fi connected/not available, GPS turnedon/off, battery life indicator, device on/off), according to someembodiments.

TABLE 8 SOFTWARE UPGRADES Primary Non- Alert Appli- Application Con-Haptic Haptic Condition cation State Mode dition Output Output SWCalendar Active Back- No Buzz- X upgrade ground event tap available SWMessages Active Back- No Buzz- X upgrade ground event tap available SWSocial Active Back- No Buzz- Pop-up upgrade Net- ground event tapavailable working SW Game Active Back- No Buzz- Pop-up upgrade groundevent tap available

Table 8 includes various software upgrade available alert conditions forfour different application types: calendar, messages, Social Networking,and Game. These alert conditions each are associated with differentapplications, but are in the same class of alert condition correspondingto functions in a predefined class of functions, in this caseapplication software upgrades for all applications. For each alertcondition, a haptic of buzz-tap is output. Some of the alert conditionsalso have a non-haptic output. In the case of social networking orgaming applications, the haptic output is complimented by a pop-up typenotification. Table 8 provides conditions and corresponding outputsaccording to some embodiments, but is merely exemplary; othercombinations of alerts, states, modes, conditions and outputs are usedaccording to different embodiments.

According to some embodiments, when the first alert condition and thesecond alert condition are different alert conditions that are in a sameclass of alert condition, the first haptic component of the first outputis identical (or substantially identical) to the second haptic componentof the second output, and the first non-haptic component (e.g., audio orvisual) of the first output is different from the second non-hapticcomponent of the second output. Each of the examples of Tables 4, 5, and6 above fall into this category, as they each share a common hapticcomponent and differing non-haptic components. In some embodiments,“identical” haptic components are haptic components that are generatedbased on same haptic output instructions delivered to a haptic outputgenerator, even if the haptic output generator (sometimes an analogcomponent) generates haptic outputs that are slightly different (evenwhile being substantially perceptually identical to a typical user). Insome embodiments, the audio components may share attributes, but stilldiffer. For example, the same ringtone or musical notes played indifferent pitches or using different musical instruments. Overlayingdifferent audio (e.g., high vs low pitch, same audio played in metallicvs. glass vs. ceramic tones) on the same haptic output creates differentperceptions. In some embodiments, different haptic components are hapticcomponents that are generated based on different haptic outputinstructions delivered to a haptic output generator so as to producehaptic outputs that are perceptually differentiable by a typical user).

According to some embodiments, the opposite correlation applies forhaptic versus non-haptic similarities and differences in theirrespective outputs. When the first alert condition and the second alertcondition are different alert conditions that are in a same class ofalert condition, the first non-haptic component of the first output isidentical (or substantially identical) to the second non-hapticcomponent of the second output, and the first haptic component of thefirst output is different from the second haptic component of the secondoutput.

When the first alert condition and the second alert condition correspondto a same alert condition, the first output is accompanied by a firstadditional component that is different from a second additionalcomponent that accompanies the second output according to someembodiments. For example, a first and a second incoming instant messageboth are incoming instant messages, and therefore are the same alertcondition. In this example, the first additional component is unique toa first event that triggered the first alert condition, and the secondadditional component is unique to a second event that triggered thesecond alert condition, e.g., the additional component of the firstincoming instant message includes a visual component (text, metadata,graphic) that is unique to the first instant message (e.g., name ofsender, contact information of the sender, body/content of the message)according to some embodiments. Likewise, the additional component of thesecond incoming instant message includes a visual component (text,metadata, graphic) that is unique to the second instant message (e.g.,name of sender, contact information of the sender, body/content of themessage). The first additional component accompanying the first outputoptionally provides information (e.g., textual information) describingthe first event that triggered the first alert (e.g., a snippet of themessage or text version of the automated assistant response), and thesecond additional component accompanying the second output providesinformation (e.g., textual information) describing the second event thattriggered the second alert according to some embodiments.

According to some embodiments, after providing 1320 the second output,the device 100 detects 1325 occurrence of a third alert condition, andresponsive to detecting the occurrence of the third alert condition,provides 1330 a third output that includes a third haptic component anda third non-haptic component. When the first alert condition, the secondalert condition, and the third alert condition are different alertconditions that are in a same class of alert condition, the firstoutput, the second output, and the third output optionally share one ormore same components and have one or more different components. When thethird alert condition is in a different class of alert condition fromthe first alert condition and the second alert condition, the thirdhaptic component optionally is different from both the first hapticcomponent and the second haptic component and the third non-hapticcomponent optionally is different from both the first non-hapticcomponent and the second non-haptic component. Alternatively, when thefirst alert condition, the second alert condition, and the third alertcondition are different alert conditions that are in the same class ofalert condition, and the first, second, and third outputs have one ormore components in common (e.g., haptic, non-haptic) and when the thirdalert condition is in a different class of alert condition from thefirst alert condition and the second alert condition, the third hapticoutput has no common components with the first output and the secondoutput, according to some embodiments.

It should be understood that the particular order in which theoperations in FIG. 13 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200) are alsoapplicable in an analogous manner to method 1300 described above withrespect to FIG. 13. For example, the inputs, alert conditions,applications, and haptic outputs described above with reference tomethod 1300 optionally have one or more of the characteristics of theinputs, alert conditions, applications, and haptic outputs describedherein with reference to other methods described herein (e.g., methods700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200). For brevity,these details are not repeated here.

The operations described above with reference to FIGS. 7, 9-13 are,optionally, implemented by components depicted in FIGS. 1A-1B, and14-19. For example, receiving input, detecting alert conditions,determining states, determining alert condition triggers, providinghaptic and audio output, and performing operations, are, optionallyimplemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub-event, such as selection of an object on a userinterface. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionallyutilizes or calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B and 14-19.

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 14 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 14 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 14, the device 100 includes a display unit 1401configured to display an application, a touch-sensitive surface unit1403 configured to receive user contacts, and a processing unit 1405coupled to the display unit 1401 and the touch-sensitive surface unit1403. In some embodiments, the processing unit 1405 includes a detectingunit 1410, a state determining unit 1415, a haptic providing unit 1420,an engagement determining unit 1425, and output determining unit 1430.

The processing unit 1405 is configured to detect an alert conditionassociated with an application executing on the computing device (e.g.,with the detecting unit 1410). The processing unit 1405 also isconfigured to determine a state associated with the application at atime associated with the alert condition responsive to detecting thealert condition (e.g., with the state determining unit 1415). Inaccordance with a determination that the application was in an activestate at the time associated with the alert condition, the processingunit 1405 provides a first haptic output (e.g., with the hapticproviding unit 1420) that represents the occurrence of the alertcondition, the first haptic output having a first set of outputcharacteristics and in accordance with a determination that theapplication was in an inactive state at the time associated with thealert condition, providing a second haptic output (e.g., with the hapticproviding unit 1420) that represents the occurrence of the alertcondition, the second haptic output having a second set of outputcharacteristics, wherein the second haptic output is different from thefirst haptic output.

The following 13 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 14:

The processing unit 1405 determines the state associated with theapplication at a time associated with the alert condition by determiningwhether a user interface for the application was displayed on atouchscreen of the device at the time associated with the alertcondition (e.g., with the state determining unit 1415).

The processing unit 1405 determines the state associated with theapplication at a time associated with the alert condition by determiningwhether a user input to the application triggered the alert condition(e.g., with the state determining unit 1415), with the user interactionbeing different from the alert condition.

If the application was in the active state based on a detected userinteraction at the time of the alert condition and the processing unit1405 determines a level of user engagement associated with the userinteraction at the time of the alert condition (e.g., with theengagement determination unit 1425) and determines one or more of thefirst set of output characteristics for the first haptic output based onthe determined level of engagement (e.g., with the output determiningunit).

The processing unit 1405 determining that the application was in theactive state comprises determining that the application was executing inthe foreground on the computing device at the time associated with thealert condition. The processing unit 1405 determining that theapplication was in the inactive state comprises determining that theapplication was not executing on the computing device or was executingin the background on the computing device at the time associated withthe alert condition.

The display unit 1401 displays a respective user interface windowcorresponding to the application concurrently with one or more otheruser interface windows on a multi-application-window user interface onthe device. The processing unit 1405 determines whether the respectiveuser interface window corresponding to the application was displayed inthe foreground of the multi-application-window user interface at thetime associated with the alert condition (e.g., with the statedetermining unit 1415).

The display unit 1401 displays one or more other user interface windowsare concurrently on a multi-application-window user interface on thedevice at the time associated with the alert condition and theprocessing unit 1405 determines the state associated with theapplication (e.g., with the state determining unit 1415) at the timeassociated with the alert condition comprises determining whether arespective user interface window corresponding to the application wasdisplayed in the multi-application-window user interface at the timeassociated with the alert condition.

In some embodiments, the alert condition corresponds to an event that isautomatically triggered by the application, an automatically initiatedevent notification received by the application from a source external tothe device, or a manually initiated event notification received by theapplication from a human user other than a user of operating the device.

The processing unit 1405 provides output (e.g., with output determiningunit 1430) comprising a first haptic output accompanied by a first audiooutput and a second haptic output accompanied by a second audio output,with the first haptic output being identical to the second haptic outputand the first audio output being different from the second audio output.Alternatively, the processing unit 1405 provides output (e.g., withoutput determining unit 1430) comprising a first haptic outputaccompanied by an audio output and the second haptic output accompaniedby the same audio output, with the first haptic output being differentfrom the second haptic output.

The processing unit 1405 accompanies the second haptic output by anaudio output, a waveform of the second haptic output generated based on,mirroring, and synchronized to a waveform of the accompanying audiooutput. The processing unit 1405 provides output (e.g., with outputdetermining unit 1430) comprising a first haptic output not accompaniedby audio output and a second haptic output accompanied by an audiooutput. In some embodiments, the first haptic output is similar to thesecond haptic output. In some embodiments, the second haptic output hasa greater intensity than the first haptic output.

The first set of characteristics of the haptic output (e.g., provided byoutput determining unit 1430) comprises one or more of: an amplitude, aduration, a regularity, a repetition frequency, or a selection of hapticfeatures of the first haptic output. In some embodiments, the secondhaptic output is different and has a greater intensity than the firsthaptic output.

The processing unit 1405 determines a state of the computing device(e.g., with the state determining unit) at the time associated with thealert condition by determining that the computing device was active atthe time associated with the alert condition, and determining that theapplication was in the inactive state comprises determining that thecomputing device was inactive at the time associated with the alertcondition. The first and second sets of characteristics correspond to adevice type of the computing device according to some embodiments.

In some embodiments, at a first point in time, while the application isin the active state, the processing unit 1405 detects a first alertcondition associated with the application (e.g., with the detecting unit1410), and responsive to detecting the first alert condition while thefirst application is in the active state, the processing unit 1404provides (e.g., with the output determining unit 1430) a first outputrepresenting the first alert condition. At a second point in time, whilethe first application is in an inactive state, the processing unit 1405detects a second alert condition, and responsive to detecting the secondalert condition while that the application is in the inactive state, theprocessing unit 1405 provides the second haptic output representing thesecond alert condition (e.g., with output determining unit 1440), thesecond haptic output different from the first haptic output.

In some embodiments, the device has a touch-sensitive display (e.g.,provided by touch-sensitive surface unit 1403), and the first hapticoutput and the second haptic output are provided via a touch-sensitivedisplay on the computing device (e.g., with output determining unit1430). As mentioned above, the operations described with reference toFIGS. 7 and 9-13 are optionally implemented by components depicted inFIGS. 14-19. FIG. 14 shows an example functional block diagram of device100 configured in accordance with the principles of the variousdescribed embodiments. It is understood by persons of skill in the artthat the functional blocks described in FIG. 14 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein.

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 15 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 15 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 15, the device 100 includes a display unit 1501configured to display an application, a touch-sensitive surface unit1503 configured to receive user contacts, and a processing unit 1505coupled to the display unit 1501 and the touch-sensitive surface unit1503. In some embodiments, the processing unit 1505 includes a detectingunit 1510, a trigger determining unit 1515, a haptic providing unit1520, an external event determining unit 1525, an output variantproviding unit 1530, an urgency determining unit 1535, a contextualdetermining unit 1540, a modulating unit 1545, and a personalcommunication determining unit 1550.

The processing unit 1505 is configured to detect occurrence of an alertcondition associated with an application (e.g., with detecting unit1510), and responsive to detecting the occurrence of the alertcondition, determine whether the alert condition was triggered by amanually-initiated event (e.g., with trigger determining unit 1515). Inaccordance with a determination that the alert condition was triggeredby a manually-initiated event, the processing unit 1505 provides a firsthaptic output (e.g., with haptic providing unit 1520) corresponding to amanually-initiated event notification in accordance with a determinationthat the alert condition was triggered by an automatically-initiatedevent, the processing unit 1505 provides a second haptic output (e.g.,with haptic providing unit 1520) corresponding to anautomatically-initiated event notification, wherein the second hapticoutput is different from the first haptic output.

The following 12 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 15:

The processing unit 1505 determines that the alert condition wastriggered by a manually-initiated event by determining whether the alertcondition corresponds to an event initiated by a human user (e.g., withtrigger determining unit 1515), according to some embodiments. In someembodiments, the processing unit 1505 determines that the alertcondition was triggered by a manually-initiated event by determiningwhether the alert condition corresponds to an event initiated by a humanuser other than a user of the computing device (e.g., with triggerdetermining unit 1515).

The processing unit 1505 determines that the alert condition wastriggered by a manually-initiated event by determining whether the alertcondition corresponds to an input received from a human user (e.g., withtrigger determining unit 1515), the input identifying a user of thecomputing device and the input comprising an indication from the humanuser to alert the user of the computing device as to the input,according to some embodiments.

The processing unit 1505 determines whether the alert condition wasautomatically triggered by the application or was anautomatically-initiated event notification received by the applicationfrom a source external to the device (e.g., with trigger determiningunit 1515), according to some embodiments. Alternatively, the processingunit 1505 determines that the alert condition was triggered by anautomatically-initiated event by determining that the alert conditionoccurs at predetermined time or reports on satisfaction of apredetermined triggering condition (e.g., with trigger determining unit1515).

In accordance with a determination that the alert condition wastriggered by an automatically-initiated event, the processing unit 1505determines whether the automatically-initiated event corresponds to anautomatically-initiated event occurring external to the device (e.g.,with external event determining unit 1525), according to someembodiments. In accordance with a determination that theautomatically-initiated event corresponds to an event occurring externalto the device, the processing unit 1505 provides a first variant of thesecond haptic output (e.g., with the output variant providing unit1530), corresponding to a notification of an externally occurringautomatically-initiated event. In accordance with a determination thatthe automatically-initiated event corresponds to an event initiatedwithin the device, the processing unit 1505 provides a second variant ofthe second haptic output corresponding to a notification of aninternally generated automatically-initiated event (e.g., with theoutput variant providing unit 1530).

In accordance with a determination that the alert condition wastriggered by a manually-initiated event, the processing unit 1505determines a measure of urgency associated with the manually-initiatedevent (e.g., with urgency determining unit 1535), and modulates anintensity of the first haptic output based on the measure of urgency(e.g., with modulating unit 1545), according to some embodiments. Inaccordance with a determination that the alert condition was triggeredby a manually-initiated event, the processing unit 1505 according tosome embodiments determines one or more contextual attributes associatedwith the manually-initiated event (e.g., with contextual determiningunit 1540), and modulates an intensity of the first haptic output basedon the one or more contextual attributes (e.g., with the modulating unit1545), according to some embodiments.

In accordance with a determination that the alert condition wastriggered by an automatically-initiated event, the processing unit 1505analyzes characteristics associated with the alert condition todetermine one or more contextual attributes associated with theautomatically-initiated event (e.g., with contextual determining unit1540), and modulates an intensity the second output based on the one ormore contextual attributes (e.g., with the modulating unit 1545),according to some embodiments.

In accordance with a determination that the alert condition wastriggered by an automatically-initiated event, the processing unit 1505determines a time-varying measure of urgency associated with theautomatically-initiated event (e.g., with urgency determining unit1535), and modulates, over a specified window of time, an intensity ofthe second haptic output to indicate the time-varying measure of urgency(e.g., with modulating unit 154), according to some embodiments.

In some embodiments, the first haptic output is accompanied by a firstaudio output and the second haptic output is accompanied by a secondaudio output, the first haptic output is similar to the second hapticoutput, and the first audio output is different from the second audiooutput. In some embodiments, the first haptic output is accompanied by afirst audio output and the second haptic output is accompanied by asecond audio output, the first audio output is similar to the secondaudio output, and the first haptic output is different from the secondhaptic output. In some embodiments, the first haptic output isaccompanied by a first audio output and the second haptic output is notaccompanies by an audio output. The first haptic output optionally issimilar to the second haptic output.

The first haptic output that corresponds to the manually-initiated eventnotification is more intense than the second haptic output thatcorresponds to the automatically-initiated event notification, accordingto some embodiments. In some circumstances, the first haptic output ischaracterized by a first set of characteristics comprising one or moreof: an amplitude of the first haptic output, a duration of the firsthaptic output, a regularity associated with the first haptic output,repetition frequency of haptic features in the first haptic output, anda selection of haptic features constituting the first haptic output, andthe second haptic output is characterized by a second set ofcharacteristics comprising one or more of: an amplitude of the secondhaptic output, a duration of the second haptic output, a regularityassociated with the second haptic output, repetition frequency of hapticfeatures in the second haptic output, and a selection of haptic featuresconstituting the second haptic output.

The processing unit 1505 detects a first alert condition correspondingto an incoming email message from a first human sender (e.g., with thedetecting unit 1510), according to some embodiments. Responsive todetermining that the incoming email message corresponds to amanually-initiated event, the processing unit 1505 provides the firsthaptic output (e.g., with haptic providing unit 1520). The processingunit 1505 detects a second alert condition corresponding to an incomingtext message from a second human sender (e.g., with the detecting unit1510), and responsive to determining that the incoming text messagecorresponds to a manually-initiated event, provides the first hapticoutput (e.g., with haptic providing unit 1520).

In some embodiments, the device has a touch-sensitive display (e.g.,provided by touch-sensitive surface unit 1503), and the first hapticoutput and the second haptic output are provided via a touch-sensitivedisplay on the computing device (e.g., with haptic providing unit 1520).

The processing unit 1505 detects occurrence of an alert conditionassociated with an application (e.g., with the detecting unit 1510),according to some embodiments, and responsive to detecting theoccurrence of the alert condition, determines whether the alertcondition corresponds to a personal communication from a sender in alist of contacts associated with the user (e.g., with personalcommunication determining unit 1550). In accordance with a determinationthat the alert condition corresponds to a personal communication from asender in a list of contacts associated with the user, the processingunit 1505 provides a first haptic output corresponding to a personalalert notification (e.g., with haptic providing unit 1520), and inaccordance with a determination that the alert condition does notcorrespond to a personal communication from a sender in a list ofcontacts associated with the user, the processing unit 1505 provides asecond haptic output corresponding to an automatic alert notification(e.g., with haptic providing unit 1520), wherein the first haptic outputis greater in magnitude than the second haptic output.

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 16 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 16 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 16, the device 100 includes a display unit 1601configured to display an application, a touch-sensitive surface unit1603 configured to receive user contacts, and a processing unit 1605coupled to the display unit 1601 and the touch-sensitive surface unit1603. In some embodiments, the processing unit 1605 includes a detectingunit 1610, an output providing unit 1615, a modulation conditiondetermining unit 1620, an output modifying unit 1625, an engagementmeasuring unit 1630, a tactile input determining unit 1635, an outputterminating unit 1640, an ambient condition determining unit 1645, anoutput delaying unit 1650, and a priming providing unit 1655.

The processing unit 1605 is configured to detect a first alert conditionon the computing device associated with receiving user input to anapplication (e.g., with the detecting unit 1610), and responsive todetecting the first alert condition, provides a first haptic outputhaving a first intensity and corresponding to the user input to theapplication (e.g., with the output providing unit 1615). After providingthe first haptic output, the processing unit 1605 detects a second alertcondition associated with receiving a predetermined system event in theapplication (e.g., with the detecting unit 1610), and responsive todetecting the second alert condition, provides a second haptic outputhaving a second intensity (e.g., with the output providing unit 1615),and corresponding to the predetermined system event, wherein the secondintensity is greater than the first intensity.

The following 18 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 16:

In some embodiments, the first haptic output is feedback provided indirect response to and as a consequence of the user input. The secondalert condition corresponds to an event that is automatically triggeredby the application, an automatically initiated event notificationreceived by the application, or a manually initiated event notificationreceived by the application, according to some embodiments.

The processing unit 1605 determines whether an input-based modulationcondition exists (e.g., with the modulation condition determining unit1620), and responsive to determining that an input-based modulationcondition exists, prior to providing the first haptic output, theprocessing unit 1605 modifies the first haptic output (e.g., with theoutput modifying unit 1625), according to some embodiments.

In some embodiments, after detecting a respective alert condition, theprocessing unit 1605 determines that the respective alert conditionresulted from a respective user input received on the computing deviceand identifies a measure of user engagement associated with the detecteduser input (e.g., with the engagement measuring unit 1630). Theprocessing unit 1605 modulates the first signal intensity of the firsthaptic output based on the measure of user engagement (e.g., with theoutput modifying unit 1625).

In some embodiments, after detecting a respective alert condition, theprocessing unit 1605 determines that the respective alert conditionresulted from receiving a tactile user input on the device (e.g., withthe tactile input determining unit 1635), and identifies a location ofreceiving the tactile user input on the device. The processing unit 1605modulates a spatial haptic energy profile of the first haptic output, byproviding greater than a specified proportion of haptic energy of thefirst haptic output within a specified radius threshold of the locationof receiving the tactile user input on the device (e.g., with the outputmodifying unit 1625).

In some embodiments, after detecting a respective alert condition, theprocessing unit 1605 determines that the respective alert conditionresulted from receiving a tactile user input on the device (e.g., withthe tactile input determining unit 1635), and modulates a spatial hapticenergy profile of the first haptic output (e.g., with the outputmodifying unit 1625), by reducing the amplitude of haptic energy of thefirst haptic output such that it is perceptible at the location ofreceiving the tactile user input on the device. The location ofreceiving the tactile user input on the device can vary with time, andthe spatial haptic energy can varies with time corresponding to thetime-varying location of receiving the tactile user input.

In some embodiments, after detecting a respective alert condition, theprocessing unit 1605 determines that the respective alert conditionresulted from receiving a tactile user input, and identifiestime-varying morphological attributes of the tactile user input (e.g.,with the tactile input determining unit 1635). The processing unit 1605modifies, over time, morphological characteristics of the first hapticoutput to mimic the time-varying morphological attributes of the tactileuser input (e.g., with the output modifying unit 1625). The time-varyingmorphological attributes include a time-dependent contact pressureprofile between the tactile user input and a touch-sensitive surface ofthe computing device, according to some embodiments, and the modifyingcomprises modulating, with time, an energy profile of the first hapticoutput to mirror the time-dependent contact pressure profile between thetactile user input and the touch-sensitive surface of the computingdevice (e.g., with the output modifying unit 1625).

In some embodiments, after detecting a respective alert condition, theprocessing unit 1605 determines that the respective alert conditionresulted from a sustained user input (e.g., with the tactile inputdetermining unit 1635), and responsive to the determining, initiates thefirst haptic output at an initiation time of the sustained user input(e.g., with the output providing unit 1615). The processing unit 1605detects termination of the sustained user input, and terminates thefirst haptic output at a termination time of the sustained user input(e.g., with the output terminating unit 1640).

The processing unit 1605 determines whether an ambient condition-basedmodulation condition exists (e.g., with the ambient conditiondetermining unit 1645), and responsive to determining that an ambientcondition-based modulation condition exists, prior to providing thesecond haptic output, modifies the second haptic output (e.g., with theoutput modifying unit 1625).

In some embodiments, after detecting a respective alert conditionassociated with a respective application, the processing unit 1605determines that the respective alert condition did not result fromreceiving user input, and responsive to determining that the respectivealert condition did not result from receiving user input, the processingunit 1605 determines whether one or more ambient conditions exist forthe device as potential interference to haptic output at a time ofoccurrence of the respective alert condition (e.g., with the ambientcondition determining unit 1645). In accordance with a determinationthat no interferential ambient conditions exist for the device, theprocessing unit 1605 provides a respective second haptic output based onthe state of receptiveness of the user (e.g., with the output providingunit 1615). In accordance with a determination that one or more ambientconditions exist for the device as potential interference to hapticoutput, the processing unit 1605 delays providing the respective secondoutput to the user (e.g., with the output delaying unit 1650).

In some embodiments, after detecting a respective alert conditionassociated with a respective application, the processing unit 1605determines that the respective alert condition did not result fromreceiving user input, and responsive to determining that the respectivealert condition did not result from receiving user input, the processingunit 1605 determines whether one or more ambient conditions exist forthe device as potential interference to haptic output at a time ofoccurrence of the respective alert condition (e.g., with the ambientcondition determining unit 1645). In accordance with a determinationthat no interferential ambient conditions exist for the device, theprocessing unit 1605 provides a first variant of the second hapticoutput to the user (e.g., with the output modifying unit 1625), thefirst variant of the second haptic output having a first set of outputcharacteristics. In accordance with a determination that one or moreambient conditions exist for the device as potential interference tohaptic output, the processing unit 1605 provides a second variant of thesecond haptic output to the user (e.g., with the output modifying unit1625), the second variant of the second haptic output having a secondset of output characteristics, the second set of characteristicsdifferent from the first set of characteristics, the second variant ofthe second haptic output having greater intensity than the first variantof the second haptic output.

In some embodiments, after detecting a respective alert conditionassociated with a respective application, the processing unit 1605determines that the respective alert condition did not result fromreceiving user input, and responsive to determining that the respectivealert condition did not result from receiving user input, the processingunit 1605 provides a priming haptic output as a precursor to the secondhaptic output (e.g., with the priming providing unit 1655), the priminghaptic output to increase an engagement level of a user operating thedevice to the second haptic output, wherein at least a portion of thepriming haptic output is more pronounced than the second haptic output.In addition, the processing unit 1605 provides the second haptic outputfollowing and within a specified time interval of providing the priminghaptic output (e.g., with the output providing unit 1615). In someembodiments, the priming haptic output comprises a priming hapticcomponent with a time-varying waveform morphology characterized by aprogressively increasing intensity of the haptic component with time. Insome embodiments, the priming haptic output comprises a priming hapticcomponent with a time-varying waveform morphology characterized by anemphasized haptic feature followed by a progressively diminishingintensity of the haptic component.

In some embodiments, the second haptic output is preceded by the priminghaptic output, and the first haptic output is provided directly, withouta corresponding preceding priming haptic output, in response to acorresponding alert condition that results from a received user input.In some embodiments, characteristics of the priming haptic output areselected based on an urgency or context associated with the alertcondition.

In some embodiments, the processing unit 1605 detects a first alertcondition corresponding to a user selection of a user interface elementdisplayed on an application user interface associated with a firstapplication (e.g., with the detecting unit 1610). Responsive to thefirst alert condition, the processing unit 1605 provides a respectivefirst haptic output representing the user selection of a user interfaceelement (e.g., with the output providing unit 1615), the respectivefirst haptic output comprising a first haptic component including afirst haptic feature of a first intensity and a first duration. Afterproviding the respective first haptic output, the processing unit 1605detects a second alert condition corresponding to an alert notificationreceived by the first application (e.g., with the detecting unit 1610),and responsive to the second alert condition, the processing unit 1605provides a respective second haptic output representing the receipt ofthe alert notification (e.g., with the output providing unit 1615), therespective second haptic output comprising a second haptic componentincluding a second haptic feature of a second intensity and a secondduration, the respective second haptic output different from and morepronounced than the respective first output based on the second hapticfeature being more pronounced than the first haptic feature, the secondintensity being greater than the first intensity, or the second durationbeing longer than the first duration.

In some embodiments, the processing unit 1605 detects a first alertcondition associated with a first application, the first alert conditionresulting from a respective user input (e.g., with the detecting unit1610), and provides a respective first haptic output representing thefirst alert condition (e.g., with the output providing unit 1615). Afterproviding the respective first haptic output, the processing unit 1605detects a second alert condition associated with the first application(e.g., with the detecting unit 1610), the second alert condition notresulting from receiving user input, and provides a respective secondhaptic output representing the second alert condition (e.g., with theoutput providing unit 1615), the respective second haptic outputdifferent from and having greater intensity than the respective firstoutput.

In some embodiments, the first haptic output has a first set ofcharacteristics comprising one or more of: an amplitude of the firsthaptic output, a duration of the first haptic output, a regularityassociated with the first haptic output, repetition frequency of hapticfeatures in the first haptic output, a selection of haptic featuresconstituting the first haptic output, and the second haptic output has asecond set of characteristics comprising one or more of: an amplitude ofthe second haptic output, a duration of the haptic component associatedwith the second haptic output, a regularity associated with the secondhaptic output, repetition frequency of haptic features in the secondhaptic output, a selection of haptic features constituting the secondhaptic output.

In some embodiments, the first haptic output is accompanied by a firstaudio output and the second haptic output is accompanied by a secondaudio output, the first haptic output is similar to the second hapticoutput, and the first audio output is different from the second audiooutput. In some embodiments, the first haptic output is accompanied by afirst audio output and the second haptic output is accompanied by asecond audio output, the first audio output is similar to the secondaudio output, and the first haptic output is different from the secondhaptic output.

In some embodiments, the first intensity of the first haptic output andthe second intensity of the second haptic output is further adjusted ininverse relation to a volume setting on the device. The second hapticoutput is accompanied by an audio output and the first haptic output isnot accompanied by an audio output, according to some embodiments.

In some embodiments, the computing device comprises a touch-sensitivedisplay for receiving the user input (e.g., with the touch-sensitivesurface unit 1603), and the first haptic output and the second hapticoutput are provided via the touch-sensitive display on the computingdevice (e.g., with the output providing unit 1615).

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 17 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 17 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 17, the device 100 includes a display unit 1701configured to display an application, a touch-sensitive surface unit1703 configured to receive user contacts, and a processing unit 1705coupled to the display unit 1701 and the touch-sensitive surface unit1703. In some embodiments, the processing unit 1705 includes an inputreceiving unit 1710, an ongoing output initiating unit 1715, an ongoingoutput termination unit 1720, a feedback providing unit 1725, and anadditional input receiving unit 1730.

The processing unit 1705 is configured to receive an input thatcorresponds to a first part of a multi-part operation performed by anapplication executing on the computing device (e.g., with the inputreceiving unit 1710), and responsive to receiving the inputcorresponding to the first part of the multi-part operation, initiate anongoing haptic output sequence (e.g., with the ongoing output initiatingunit 1715). After initiating the ongoing haptic output sequence, theprocessing unit 1705 receives an input on the that corresponds to asecond part of the multi-part operation (e.g., with the input receivingunit 1710), and responsive to receiving the input corresponding to thesecond part of the multi-part operation, terminates the ongoing hapticoutput sequence (e.g., with the ongoing output termination unit 1720).

The following 16 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 17:

In some embodiments, the input corresponding to the first part of themulti-part operation is detected on a touch-sensitive surface of thedevice (e.g., with the touch-sensitive surface unit 1703), and the inputcorresponding to the second part of the multi-part operation is detectedon the touch-sensitive surface of the device (e.g., with the inputreceiving unit 1710). The haptic output sequence is provided via thetouch-sensitive surface (e.g., with the ongoing output initiating unit1715), according to some embodiments.

In some embodiments, the input corresponding to the first part of themulti-part operation is detected by the device (e.g., with the inputreceiving unit 1710), and the input corresponding to the second part ofthe multi-part operation detected by a second device. In someembodiments, the input corresponding to the first part of the multi-partoperation and the input corresponding to the second part of themulti-part operation is detected by the device (e.g., with the inputreceiving unit 1710).

According to some embodiments, the multi-part operation is a securetransaction, the first part of the multi-part operation includes armingthe device to authorize the secure transaction, and the second part themulti-part operation includes authorizing the secure transaction.According to some embodiments, the multi-part operation is a securetransaction, the first part of the multi-part operation includesdetermining that user authentication is required to complete the securetransaction, and the second part the multi-part operation includesreceiving user authentication for the secure transaction.

In some embodiments, the first part of the multi-part operation includesclosing a draft of a document, and the second part of the multi-partoperation includes returning to the draft of the document. In someembodiments, the input corresponding to the first part of the multi-partoperation is a first user input interacting with an applicationexecuting on the device, and the input corresponding to the second partof the multi-part operation is a second user input interacting with theapplication, the second user input distinct from the first user input.

According to some embodiments, detecting the input that corresponds tothe first part of the multi-part operation includes detecting a contacton the touch-sensitive surface (e.g., with the input receiving unit1710), and detecting the input that corresponds to the second part ofthe multi-part operation includes detecting movement of the contact onthe touch-sensitive surface (e.g., with the input receiving unit 1710).In some embodiments, detecting the input that corresponds to the firstpart of the multi-part operation includes detecting movement of acontact on the touch-sensitive surface (e.g., with the input receivingunit 1710), and detecting the input that corresponds to the second partof the multi-part operation includes detecting liftoff of the contactfrom the touch-sensitive surface (e.g., with the input receiving unit1710).

In some embodiments, the inputs corresponding to the first and secondparts of the multi-part operation comprise a single gesture, wherein theinput corresponding to the first part of the multi-part operation is aninitial part of the gesture, and the input corresponding to the secondpart of the multi-part operation is a subsequent part of the gesture.According to some embodiments, the gesture is initiated on a portion ofthe touch-sensitive surface that corresponds to a first location in auser interface displayed on a display of the device and terminates on asecond portion of the touch-sensitive surface that corresponds to asecond location in the user interface that is different from the firstlocation (e.g., with the touch-sensitive surface unit 1703).

In some embodiments, the input corresponding to the first part of themulti-part operation initiates an event associated with the respectiveapplication, and the event is sustained until receipt of the inputcorresponding to the second part of the multi-part operation, and theinput corresponding to the second part of the multi-part operationterminates the event associated with the respective application (e.g.,with the ongoing output termination unit 1720).

According to some embodiments, the application is a text editingapplication, and the input corresponding to the first part of themulti-part operation is a selection of a first user interface elementthat enables a text-entry mode of the text editing application. Themulti-part operation comprises one or more inputs for text-entry to thetext editing application and the event corresponds to text manipulationin the text editing document responsive to the inputs for text-entry,according to some embodiments. The input corresponding to the secondpart of the multi-part operation is a selection of a user interfaceelement that disables the text-entry mode of the text editingapplication.

In some embodiments, the input corresponding to the first part of themulti-part operation is a selection of a first user interface elementthat initiates a transaction, and the multi-part operation comprises oneor more inputs for entry of information required to process thetransaction. The event in some embodiments corresponds to population ofone or more data entry fields that receive the information required toprocess the transaction, and the input corresponding to the second partof the multi-part operation is a selection of a user interface elementthat authorizes completion of the transaction.

In some embodiments, the input corresponding to the second part of themulti-part operation follows the input corresponding to the first partof the multi-part operation and is separated in time from the inputcorresponding to the first part of the multi-part operation by aspecified interval of time corresponding to a duration of the event, andthe ongoing haptic output sequence is provided throughout the durationof the event and indicates occurrence of the event. Characteristics ofthe ongoing haptic output sequence optionally are selected based on theevent that is initiated by the input corresponding to the first part ofthe multi-part operation.

In some embodiments, the processing unit 1705 provides a first hapticfeedback responsive to receiving the input corresponding to the firstpart of the multi-part operation (e.g., with the feedback providing unit1725), the first haptic feedback selected based on the event andindicating an initiation of the event, the first haptic feedbackdistinct from the ongoing haptic output sequence. The processing unit1705 further provides a second haptic feedback responsive to receivingthe input corresponding to the second part of the multi-part operation(e.g., with the feedback providing unit 1725), the second hapticfeedback distinct from than the ongoing haptic output sequence.

The input corresponding to the first part of the multi-part operationsuspends an ongoing event associated with the application, and the inputcorresponding to the second part of the multi-part operation resumes theongoing event associated with the application, according to someembodiments.

In some embodiments, the application is a voice communicationapplication, and the event is an ongoing voice communication occurringat a time of receipt of the input corresponding to the first part of themulti-part operation. In some embodiments, the input corresponding tothe first part of the multi-part operation is a selection of a firstuser interface element of the voice communication application, theselection suspending the ongoing voice communication, and the inputcorresponding to the second part of the multi-part operation is aselection of a second user interface element that resumes the voicecommunication.

In some embodiments, after initiating an ongoing haptic output sequenceresponsive to the first part of a multi-part operation and beforereceiving an input that corresponds to a second part of the multi-partoperation, the processing unit 1705 receives one or more additional userinputs (e.g., with the additional input receiving unit 1730), distinctfrom the input that corresponds to the second part of the multi-partoperation. The processing unit 1705 continues to provide the ongoinghaptic output regardless of the additional user inputs (e.g., with theongoing output initiating unit 1715), until receipt of the inputcorresponding to the second part of the multi-part operation. In someembodiments, the intensity of the ongoing haptic output sequenceincreases with time until a subsequent input corresponding to themulti-part operation is received.

According to some embodiments, responsive to detecting the inputcorresponding to the first part of the multi-part operation (e.g., withthe input receiving unit 1710), the processing unit 1705 initiates anongoing audio output sequence to accompany the ongoing haptic sequence(e.g., with the ongoing output initiating unit 1715), and responsive todetecting the input corresponding to the second part of the multi-partoperation (e.g., with the input receiving unit 1710), terminating theongoing the audio output sequence (e.g., with the ongoing outputtermination unit 1720).

In some embodiments, a time-varying haptic energy profile of the hapticoutput sequence mimics a time-varying acoustic energy profile of theaudio output sequence. In some circumstances, responsive to determiningthat the input corresponding to the first part of the multi-partoperation does not pause an ongoing event, but initiates an eventassociated with the respective application, the processing unit 1705provides the haptic output sequence without an accompanying audio outputsequence (e.g., with the ongoing output initiating unit 1715), andresponsive to determining that the input corresponding to the first partof the multi-part operation pauses an ongoing event associated with therespective application, the processing unit 1705 provides an ongoingaudio output sequence accompanying the ongoing haptic output sequence(e.g., with the ongoing output initiating unit 1715).

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 18 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 18 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 18, the device 100 includes a display unit 1801configured to display an application, a touch-sensitive surface unit1803 configured to receive user contacts, and a processing unit 1805coupled to the display unit 1801 and the touch-sensitive surface unit1803. In some embodiments, the processing unit 1805 includes an inputdetecting unit 1810, an output providing unit 1815, an operationperforming unit 1820, an image capturing unit 1825, a secure transactionauthorization unit 1830, and a save operation performing unit 1835.

The processing unit 1805 is configured to detect a first input thatcorresponds to a request to perform a first operation (e.g., with theinput detecting unit 1810), and responsive to detecting the first input,to provide a first output that includes a haptic component (e.g., withthe output providing unit 1815). Also responsive to detecting the firstinput, the processing unit 1805 is configured to perform the firstoperation (e.g., with the operation performing unit 1820). Afterperforming the first operation, the processing unit 1805 is configuredto detect a second input that corresponds to a request to perform asecond operation that includes the first operation and a supplementaloperation (e.g., with the input detecting unit 1810), and responsive todetecting the second input, to provide a second output that includes ahaptic component (e.g., with the output providing unit 1815), whereinthe second output includes the first output in conjunction withproviding supplemental output that corresponds to the supplementaloperation. Also responsive to detecting the second input, the processingunit 1805 is configured to perform the second operation (e.g., with theoperation performing unit 1820).

The following 10 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 18:

In some embodiments, responsive to detecting the second input theprocessing unit 1805 performs the first operation and concurrentlyprovides the first output, and performs the supplemental operation andconcurrently provides the supplemental output (e.g., with the operationperforming unit 1820 and the output providing unit 1815).

In some embodiments, responsive to detecting the second input, theprocessing unit 1805 performs the first operation prior to performingthe supplemental operation (e.g., with the operation performing unit1820), and provides the first output prior to providing the supplementaloutput (e.g., with the output providing unit 1815). In some embodiments,responsive to detecting the second input, the processing unit 1805performs the first operation after performing the supplemental operation(e.g., with the operation performing unit 1820), and provides the firstoutput after providing the supplemental output (e.g., with the outputproviding unit 1815).

In some embodiments, responsive to detecting the second input, theprocessing unit 1805 performs at least a portion of the first operationconcurrently with a portion of the supplemental operation (e.g., withthe operation performing unit 1820), and provides at least a portion ofthe first output concurrently with a portion of the supplemental output(e.g., with the output providing unit 1815).

In some embodiments, the first operation corresponds to capture of animage with a camera, and the second operation corresponds to capturingan image after a specified interval of time. In some embodiments, thefirst operation corresponds to a transaction-enable operation forenabling the device to authorize a secure transaction, and the secondoperation corresponds to enabling the device to authorize a securetransaction and authorizing the secure transaction.

According to some embodiments, the first operation corresponds to a saveoperation for saving content within an existing file, and the secondoperation corresponds to a save-as operation to save content within anexisting file to a new file. According to some embodiments, the firstoperation corresponds to a send operation for sending a reply to amessage that is in a message inbox, and the second operation correspondsto a send and archive operation for sending a reply to a message that isin the message inbox and removing it from the message inbox. In someembodiments, the supplemental operation has a variable value, and thesupplemental output is based on the value of the variable attribute.

In some embodiments, the haptic component corresponding to the secondoutput comprises a first haptic component corresponding to the firstoutput and a second haptic component corresponding to the supplementaloutput, the second haptic component is distinct from the first hapticcomponent. In some circumstances, the supplemental output comprises anon-haptic component. In some embodiments, the second output comprises atextual component that identifies the supplemental operation asdifferent from the first operation. And in some cases the supplementaloutput comprises an audio component and the first output does notinclude an audio component.

According to some embodiments, the computing device comprises atouch-sensitive display and the first and second inputs are receivedthrough the touch-sensitive display (e.g., with the touch-sensitivesurface unit 1803). In some embodiments, the haptic component of thefirst output and the haptic component of the second output are providedvia the touch-sensitive display (e.g., with the output providing unit1815).

In some embodiments, the processing unit 1805 detects a first input thatcorresponds to a request to capture of a first image with a camera(e.g., with the input detecting unit 1810), and responsive to detectingthe first input, the processing unit 1805 provides a first output thatincludes a haptic component (e.g., with the output providing unit 1815).Also in response to detecting the first input, the processing unit 1805captures the first image (e.g., with the image capturing unit 1825), andafter capturing the first image, detects a second input that correspondsto a request to capture a second image after a specified interval oftime (e.g., with the input detecting unit 1810). Responsive to detectingthe second input, the processing unit 1805 provides a second output thatincludes a haptic component (e.g., with the output providing unit 1815),wherein the second output includes the first output in conjunction withproviding supplemental output that corresponds to measuring lapse of thespecified interval of time, and the processing unit 1805 captures thesecond image after the specified interval of time (e.g., with the imagecapturing unit 1825).

In some embodiments, processing unit 1805 detects a first input thatcorresponds to a request to perform a transaction-enable operation forenabling the device to authorize a secure transaction (e.g., with theinput detecting unit 1810), and responsive to detecting the first input,provides a first output that includes a haptic component (e.g., with theoutput providing unit 1815). Also responsive to detecting the firstinput, the processing unit 1805 enables the device to authorize a securetransaction (e.g., with the secure transaction authorization unit 1830).After performing the transaction-enable operation, the processing unit1805 detects a second input that corresponds to a request to authorize asecure transaction (e.g., with the input detecting unit 1810), andresponsive to detecting the second input, provides a second output thatincludes a haptic component (e.g., with the output providing unit 1815),wherein the second output includes the first output in conjunction withproviding supplemental output that corresponds to authorizing the securetransaction. Also responsive to detecting the second input, theprocessing unit 1805 authorizes the secure transaction (e.g., with thesecure transaction authorization unit 1830).

In some embodiments, the processing unit 1805 detects a first input thatcorresponds to a request to perform a save operation (e.g., with theinput detecting unit 1810), and responsive to detecting the first input,provides a first output that includes a haptic component (e.g., with theoutput providing unit 1815). Also responsive to detecting the firstinput, the processing unit 1805 performs a save operation for savingcontent within an existing file (e.g., with the save operationperforming unit 1835), and after performing the save operation, theprocessing unit 1805 detects a second input that corresponds to arequest to perform a save as operation to save content within anexisting file to a new file (e.g., with the input detecting unit 1810),and responsive to detecting the second input, provides a second outputthat includes a haptic component (e.g., with the output providing unit1815), wherein the second output includes the first output inconjunction with providing supplemental output that corresponds to acreation of the new file. Also responsive to detecting the second input,the processing unit 1805 performs the save as operation (e.g., with thesave operation performing unit 1835).

As mentioned above, the operations described with reference to FIGS. 7and 9-13 are optionally implemented by components depicted in FIGS.14-19. FIG. 19 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 19 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 19, the device 100 includes a display unit 1901configured to display an application, a touch-sensitive surface unit1903 configured to receive user contacts, and a processing unit 1905coupled to the display unit 1901 and the touch-sensitive surface unit1903. In some embodiments, the processing unit 1905 includes an alertcondition detecting unit 1910 and an output providing unit 1915.

The processing unit 1905 is configured to detect occurrence of a firstalert condition (e.g., with the alert condition detecting unit 1910),and responsive to detecting the occurrence of the first alert condition,provide a first output that includes a first haptic component and afirst non-haptic component (e.g., with the output providing unit 1915).After providing the first output, the processing unit 1905 is configuredto detect occurrence of a second alert condition (e.g., with the alertcondition detecting unit 1910), and responsive to detecting theoccurrence of the second alert condition, providing a second output thatincludes a second haptic component and a second non-haptic component(e.g., with the output providing unit 1915). If the first alertcondition and the second alert condition are different alert conditionsthat are in a same class of alert condition, the first output and thesecond output share one or more same components and have one or moredifferent components, and if the first alert condition and the secondalert condition are different alert conditions that are in differentclasses of alert condition, the first haptic component is different fromthe second haptic component and the first non-haptic component isdifferent from the second non-haptic component.

The following 7 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 19:

In some embodiments, the first alert condition and the second alertcondition are in the same class of alert condition, wherein the class isassociated with alert conditions for an application. In someembodiments, the first alert condition and the second alert conditionare in the same class of alert condition, the class associated with acategory of applications. In some embodiments, the first alert conditionand the second alert condition are associated with two differentapplications, wherein the first alert condition and the second alertcondition are in the same class of alert condition, the classcorresponding to functions in a predefined class of functions within thetwo different applications. According to some embodiments, the firstalert condition and the second alert condition are associated with acommon application, wherein the first alert condition and the secondalert condition are in the same class of alert condition, the classcorresponding to functions in a predefined class of functions.

In some embodiments, the classes of alert conditions include messages,activity alerts, media playback alerts, virtual assistant alerts, systemalerts, scheduling reminders, and internet browser updates. In someembodiments, the first alert condition and the second alert conditioncorrespond to separate instances of the same alert condition, the firstoutput is identical to the second output. According to some embodiments,the first and second non-haptic components are audio outputs. Accordingto some embodiments, the first and second non-haptic components arevisual outputs.

In some embodiments, when the first alert condition and the second alertcondition are different alert conditions that are in a same class ofalert condition, the first haptic component of the first output isidentical to the second haptic component of the second output, and thefirst non-haptic component of the first output is different from thesecond non-haptic component of the second output. In some embodiments,when the first alert condition and the second alert condition aredifferent alert conditions that are in a same class of alert condition,the first non-haptic component of the first output is identical to thesecond non-haptic component of the second output, and the first hapticcomponent of the first output is different from the second hapticcomponent of the second output.

According to some embodiments, the first alert condition and the secondalert condition correspond to a same alert condition, and the firstoutput is accompanied by a first additional component that is differentfrom a second additional component that accompanies the second output.

In some embodiments, the first additional component accompanying thefirst output provides information describing the first event thattriggered the first alert, and the second additional componentaccompanying the second output provides information describing thesecond event that triggered the second alert.

In some embodiments, after providing the second output, the processingunit 1905 detects occurrence of a third alert condition (e.g., with thealert condition detecting unit 1910), and responsive to detecting theoccurrence of the third alert condition, provides a third output thatincludes a third haptic component and a third non-haptic component(e.g., with the output providing unit 1915), wherein when the firstalert condition, the second alert condition, and the third alertcondition are different alert conditions that are in a same class ofalert condition, the first output, the second output, and the thirdoutput share one or more same components and have one or more differentcomponents, and when the third alert condition is in a different classof alert condition from the first alert condition and the second alertcondition, the third haptic component is different from both the firsthaptic component and the second haptic component and the thirdnon-haptic component is different from both the first non-hapticcomponent and the second non-haptic component.

In some embodiments, the computing device has a touch-sensitive display(e.g., with the touch-sensitive surface unit) and the first hapticcomponent of the first output and the second haptic component of thesecond output are provided via the touch-sensitive display on thecomputing device (e.g., with the output providing unit 1915).

Hardware Specifications

In certain embodiments, the device 100 may include one or more acousticmodules 2006 for transmitting and receiving acoustic energy, shown insimplified schematic form. The acoustic module 2006 may be a speaker ormicrophone of an electronic device, as non-limiting examples.

As shown in FIG. 20, the device housing includes a first acoustic port2020 that is coupled to the acoustic module 2006. In some instances, theacoustic module 2006 is configured to function as either a microphone ora speaker element for the device 100. However, for the purposes of thefollowing description, the acoustic module 2006 is described as aspeaker element or module. The acoustic port 2020 includes orifices2031, 2032 that facilitate the transmission of audible signals fromacoustic module 2006 to the user's ear. In this example, the orificesextend through the housing 101 and acoustically connect internalcomponents of the acoustic module with the external environment. Inother examples, a single acoustic port may include multiple orifices ora single orifice. In some embodiments, the acoustic port 2020 may alsoinclude a housing structure 2021 (“umbrella”), screen mesh 2015 or otherprotective element configured to inhibit ingress of liquid or otherforeign matter.

The umbrella structure 2021 generally prevents water or fluid 2001 fromdirectly impacting the acoustic module 2006. In one example, theacoustic port 2001 includes one or more orifices or openings 2031, 2032that are offset with respect to an acoustic chamber of the module (e.g.,the space between umbrella 2021 and mesh 2015 and including orifices2031, 2032). The umbrella structure may be formed into the outer surfaceof the case or housing 101. For example, the umbrella structure 2021 maybe formed from the material located between two or more orifices oropenings 2031, 2032 that are offset with respect to the acousticchamber.

Further, certain embodiments may shape the acoustic chamber to reducethe likelihood that water accumulates in the chamber or other portion ofthe orifice. A screen 2015 may separate the acoustic cavity 2011 fromthe external environment and may impede the ingress of liquids or otherforeign material from the external environment into the acoustic cavity2011. For example, the sidewalls of the orifice(s) 2031, 2032 may extendsubstantially to the surface of the screen 2015 in order to reducesurface tension of water in the orifice or chamber. Likewise, anexterior of the orifice may be chamfered to reduce surface tension of aliquid adjacent the orifice and thus facilitate liquid removal.

In the present example depicted in FIG. 20, the acoustic module 2006 isa speaker module. As shown in FIG. 20, a speaker acoustic moduleincludes various components for producing and transmitting sound,including a diaphragm 2010, a voice coil 2009, a center magnet 2008, andside magnets/coils 2007. In a typical implementation, the diaphragm 2010is configured to produce sound waves or an acoustic signal in responseto a stimulus signal in the voice coil 2009. That is, a modulatedstimulus signal in the voice coil 2009 causes movement of the diaphragm2010. Movement of the diaphragm 2010 creates sound waves, whichpropagate through the acoustic cavity 2011 of acoustic module 2006 andeventually out the port 2020 to the exterior environment. In some cases,the acoustic cavity 2011 functions as an acoustical resonator having ashape and size that is configured to amplify and/or dampen sound wavesproduced by movement of the diaphragm 2010.

In certain embodiments, the acoustic module 2006 also includesstructural support elements, such as an end wall 2013 and base 2014.These elements may provide physical support for the speaker elements;various yokes, connectors and the like may also provide such support.Certain embodiments may include a gasket 2016 to seal the interior ofthe device against the environment. It should be appreciated that thestructure recited herein is meant as an example and not a limitation.For example, in alternative embodiments, the acoustic cavity may beformed from additional components or may be formed from a singlecomponent.

The acoustic module 2006 depicted in FIG. 20 is provided as one exampleof a type of speaker acoustic module; some embodiments may use differenttypes of speakers, microphones and the like. Further, although describedin the context of a speaker, the foregoing is equally applicable to amicrophone and many embodiments may likewise incorporate a microphone.

Embodiments described herein may include one or more sensors fordetecting the location of a touch. In particular, a touch sensor may bedisposed relative to the display of a wearable electronic device to forma touch-sensitive surface. The following description is provided withrespect to individual touch sensors that may be used to determine thelocation of a touch.

In one embodiment, the device may include self-capacitive touch sensorsthat may be formed from an array of self-capacitive pixels orelectrodes. FIG. 21A depicts an example touch sensor circuitcorresponding to a self-capacitance touch pixel electrode and sensingcircuit. Touch sensor circuit 2109 can have a touch pixel electrode 2102with an inherent self-capacitance to ground associated with it, and alsoan additional self-capacitance to ground that can be formed when anobject, such as finger 2112, is in proximity to or touching the touchpixel electrode 2102. The total self-capacitance to ground of touchpixel electrode 2102 can be illustrated as capacitance 2104. Touch pixelelectrode 2102 can be coupled to sensing circuit 2114. Sensing circuit2114 can include an operational amplifier 2108, feedback resistor 2116,feedback capacitor 2110 and an input voltage source 2106, although otherconfigurations can be employed. For example, feedback resistor 2116 canbe replaced by a switch capacitor resistor. Touch pixel electrode 2102can be coupled to the inverting input of operational amplifier 2108. AnAC voltage source 2106 can be coupled to the non-inverting input ofoperational amplifier 2108. Touch sensor circuit 2109 can be configuredto sense changes in the total self-capacitance 2104 of touch pixelelectrode 2102 induced by finger 2112 either touching or in proximity tothe touch sensor panel. Output 2120 can be used by a processor todetermine a presence of a proximity or touch event, or the output can beinputted into a discreet logic network to determine the presence of atouch or proximity event.

FIG. 21B depicts an example self-capacitance touch sensor 2130. Touchsensor 2130 can include a plurality of touch pixel electrodes 2102disposed on a surface and coupled to sense channels in a touchcontroller, can be driven by stimulation signals from the sense channelsthrough drive/sense interface 2125, and can be sensed by the sensechannels through the drive/sense interface 2125 as well. After the touchcontroller has determined an amount of touch detected at each touchpixel electrode 2102, the pattern of touch pixels in the touch screenpanel at which touch occurred can be thought of as an “image” of touch(e.g., a pattern of objects touching or in close proximity to the touchscreen). The arrangement of the touch pixel electrodes 2102 in FIG. 21Bis provided as one example, however, the arrangement and/or the geometryof the touch pixel electrodes may vary depending on the embodiment.

Certain embodiments may employ different touch-sensing mechanisms. Forexample, certain embodiments may use a mutual-capacitance sensor arrayto detect a touch. Some embodiments may use a resistive touch sensor, anoptical touch sensor, or any combination of any touch-sensing technologydiscussed herein. Accordingly, although self-capacitive touch sensing isdescribed in some detail, it should be understood and appreciated thatreferences herein to touch sensing and the like are intended to coverthe detection of a touch regardless of the particular type of touchsensor used in any given embodiment.

In certain embodiments, an electronic device may include one or morehaptic modules for providing haptic feedback to the user. Theembodiments described herein may relate to or take the form of one ormore haptic actuators suitable to provide perceivable haptic feedback.Such actuators may include an electromagnetic coil, a permanent magnetor other magnetic field source. The magnetic field may induce motion ina mass of the haptic actuator by exerting a Lorentz force on the masswhen the coil is energized. A direction of current through the coildetermines the direction of motion of the mass, while the strength ofthe magnetic field determines the velocity of the mass and thus themagnitude of the haptic output.

In general, haptic actuators implemented in some embodiments may beconfigured to maximize or enhance resultant mechanical energy, given avery compact form factor of the electronic device.

In one embodiment and as shown in FIGS. 22A-22B, the haptic actuator2450 may have a mass at least partially disposed within the coil 2200when the mass is in a rest state. This mass may include two magnets2211, 2212 of opposing polarities implemented as a magnet array affixedwithin a frame 2260; the frame 2260 may provide extra weight to the massand thus a stronger haptic output may be generated. A center block 2270may separate the magnets, although this is optional. A shaft 2250 mayextend through the mass such that the mass may freely slide on theshaft.

The magnet array (formed from the magnets 2211, 2212 and center block2270) may generate a radial magnetic field (see FIG. 22B) that interactswith the magnetic field of the coil 2200 when the coil is energized by acurrent. The Lorentz force resulting from the interaction of themagnetic fields causes the mass to move along the shaft 2250 in a firstdirection. Reversing current flow through the coil 2200 reverses theLorentz force. As a result, the magnetic field or force on the magnets2211, 2212 is also reversed and the mass may move in a second direction.Thus, mass may move in both directions along the shaft, depending on thedirection of current flow through the coil. Passing an alternatingcurrent through the coil 2200 may cause the central magnet array to moveback and forth along the shaft. This motion may induce a motion in thehousing 2220 of the actuator, which in turn may induce a motion in thehousing of the electronic device.

In order to prevent the central magnet array from being attracted to theshaft 2250, which could increase friction between the two and therebyincrease the force necessary to move the central magnet array and frame2260, the shaft may be formed from a non-ferritic material such astungsten, titanium, stainless steel, or the like, and the mass may slidealong the shaft on bearings.

The actuator also may have structures that provide restoring force tothe mass. For example, a spring 2210 may be located at either end of theshaft. As the mass impacts the spring, the spring compresses and storeskinetic energy. This kinetic energy may be released to return the massalong the shaft 2250, thereby sending it to or near its initial startingposition. The kinetic energy in the spring(s) may cooperate with thecoil to move the magnet in such a fashion.

Although a linear actuator has been described herein, it should beappreciated that other types of actuators may be used in differentembodiments. For example, some embodiments may employ a rotary actuator,a piezoelectric actuator, or any other suitable linear or non-linearactuator. Likewise, certain embodiments may employ multiple actuatorsworking in concert.

Embodiments described herein may take the form of, be incorporated in,or operate with a suitable electronic device. One example of such adevice is shown in FIG. 23A and takes the form of a wearable mechanism.As shown, the mechanism may be worn on a user's wrist and securedthereto by a band. The mechanism may have a variety of functionsincluding, but not limited to: keeping time; monitoring a user'sphysiological signals and providing health-related information based onthose signals; communicating (in a wired or wireless fashion) with otherelectronic devices, which may be different types of devices havingdifferent functionalities; providing alerts to a user, which may includeaudio, haptic, visual and/or other sensory output, any or all of whichmay be synchronized with one another; visually depicting data on adisplay; gather data from one or more sensors that may be used toinitiate, control, or modify operations of the device; determine alocation of a touch on a surface of the device and/or an amount of forceexerted on the device, and use either or both as input; accepting voiceinput to control one or more functions; accepting tactile input tocontrol one or more functions; and so on.

Alternative embodiments of suitable electronic devices include a phone,a tablet computing device, a portable media player, or other portablemultifunction device 100 as shown in FIG. 4A. Still other suitableelectronic devices may include laptop/notebook computers, personaldigital assistants, touch screens, input-sensitive pads or surfaces, andso on.

FIG. 24 depicts an example schematic diagram of a wearable electronicdevice. As shown in FIG. 24, the device 100 includes one or moreprocessing units 2405 that are configured to access a memory 2410 havinginstructions stored thereon. The instructions or computer programs maybe configured to perform one or more of the operations or functionsdescribed with respect to the device 100. For example, the instructionsmay be configured to control or coordinate the operation of the variouscomponents of the device. Such components include, but are not limitedto, display 2415, one or more input/output components 2420, one or morecommunication channels 2425, one or more sensors 2430, 2435, a speaker2440, microphone 2445, and/or one or more haptic feedback devices 2450.In some embodiments the speaker and microphone may be combined into asingle unit and/or may share a common port through a housing of thedevice. Some embodiments also include a battery 2455 or wireless power2460. Each of the components of the device 100 as depicted in FIG. 24can instead be a component in another figure with a corresponding nameor function.

The processing units 2405 of FIG. 24 may be implemented as anyelectronic device capable of processing, receiving, or transmitting dataor instructions. For example, the processing units 2405 may include oneor more of: a microprocessor, a central processing unit (CPU), anapplication-specific integrated circuit (ASIC), a digital signalprocessor (DSP), or combinations of such devices. As described herein,the term “processor” is meant to encompass a single processor orprocessing unit, multiple processors, multiple processing units, orother suitably configured computing element or elements.

In some embodiments the electronic device may accept a variety of bands,straps, or other retention mechanisms (collectively, “bands”). Thesebands may be removably connected to the electronic device by a lug thatis accepted in a recess or other aperture within the device and locksthereto. The lug may be part of the band or may be separable (and/orseparate) from the band. Generally, the lug may lock into the electronicdevice's recess and thereby maintain connection between the band anddevice. The user may release a locking mechanism to permit the lug toslide or otherwise move out of the recess. In some embodiments, therecess may be formed in the band and the lug may be affixed orincorporated into the device.

A user may change combinations of bands and electronic devices, therebypermitting mixing and matching of the two categories. It should beappreciated that devices having other forms and/or functions may includesimilar recesses and may releasably mate with a lug and/or bandincorporating a lug. In this fashion, an ecosystem of bands and devicesmay be envisioned, each of which is compatible with another. A singleband may be used to connect to devices, as one further example; in suchembodiments the band may include electrical interconnections that permitthe two devices to transmit signals to one another and thereby interactwith one another.

In many embodiments, the electronic device may keep and display time,essentially functioning as a wristwatch among other things. Time may bedisplayed in an analog or digital format, depending on the device, itssettings, and (in some cases) a user's preferences. Typically, time isdisplayed on a digital display stack forming part of the exterior of thedevice.

The display stack may include a cover element, such as a cover glass,overlying a display. The cover glass need not necessarily be formed fromglass, although that is an option; it may be formed from sapphire,zirconia, alumina, chemically strengthened glass, hardened plastic andso on. Likewise, the display may be a liquid crystal display, an organiclight-emitting diode display, or any other suitable display technology.Among other elements, the display stack may include a backlight in someembodiments.

The device also may comprise one or more touch sensors to determine alocation of a touch on the cover glass. A touch sensor may beincorporated into or on the display stack in order to determine alocation of a touch. The touch sensor may be self-capacitive in certainembodiments, mutual-capacitive in others, or a combination thereof.

Similarly, the device may include a force sensor to determine an amountof force applied to the cover glass. The force sensor may be acapacitive sensor in some embodiments and a strain sensor in someembodiments. In either embodiment, the force sensor is generallytransparent and made form transparent materials, or is located beneathor away from the display in order not to interfere with the view of thedisplay. The force sensor may, for example, take the form of twocapacitive plates separated by silicone or another deformable material.As the capacitive plates move closer together under an external force,the change in capacitance may be measured and a value of the externalforce correlated from the capacitance change. Further, by comparingrelative capacitance changes from multiple points on the force sensor,or from multiple force sensors, a location or locations at which forceis exerted may be determined. In one embodiment the force sensor maytake the form of a gasket extending beneath the periphery of thedisplay. The gasket may be segmented or unitary, depending on theembodiment.

The electronic device may also provide alerts to a user. An alert may begenerated in response to: a change in status of the device (one exampleof which is power running low); receipt of information by the device(such as receiving a message); communications between the device andanother mechanism/device (such as a second type of device informing thedevice that a message is waiting or communication is in progress); anoperational state of an application (such as, as part of a game, or whena calendar appointment is imminent) or the operating system (such aswhen the device powers on or shuts down); and so on. The number andtypes of triggers for an alert are various and far-ranging.

The alert may be auditory, visual, haptic, or a combination thereof. Ahaptic actuator may be housed within the device and may move linearly togenerate haptic output (although in alternative embodiments the hapticactuator may be rotary or any other type). A speaker may provideauditory components of an alert and the aforementioned display mayprovide visual alert components. In some embodiments a dedicated light,display, or other visual output component may be used as part of analert.

The auditory, haptic and/or visual components of the alert may besynchronized to provide an overall experience to a user. One or morecomponents may be delayed relative to other components to create adesired synchronization between them. The components may be synchronizedso that they are perceived substantially simultaneously; as one example,a haptic output may be initiated slightly before an auditory outputsince the haptic output may take longer to be perceived than the audio.As another example, a haptic output (or portion thereof) may beinitiated substantially before the auditory output but at a weak or evensubliminal level, thereby priming the wearer to receive the auditoryoutput.

The example electronic device may communicate with other electronicdevices either through a wired connection or wirelessly. Data may bepassed between devices, permitting one device to relay information toanother; control another; employ another's sensors, outputs, and/orinputs; and so on. FIG. 23B depicts a user 2310 wearing a sampleelectronic device 100 with a second electronic device 2320 in hispocket. The second device optionally has any of the capabilitiesdescribed herein for devices 100 and 300. Data may be wirelesslytransmitted between the electronic devices 100, 2320, thereby permittingthe user 210 to receive, view, and interact with data from the seconddevice 2320 by means of the first electronic device 100. Thus, the user2310 may have access to part or all of the second device's functionalitythrough the first electronic device 100 without actually needing tointeract directly with the second device.

Further, the electronic devices 100, 2320 may cooperate not only toshare data but to share functionality as well. For example, one of thetwo devices may incorporate a sensor, application, or function that theother lacks. The electronic device lacking such capabilities may requestthem from the other device, which may share wirelessly with therequesting device. Thus, multiple devices may operate together toprovide expanded functions, software, access and the like between thetwo and ultimately to a user. As one non-limiting example, theelectronic device 100 may be unable to place or receive telephone callswhile the second device 2320 may be able to do so. A user maynonetheless make and/or receive calls through the first device 100,which may employ the second device 2320 to actually place or accept acall.

As another non-limiting example, an electronic device 100 may wirelesslycommunicate with a sales terminal nearby, thus permitting a user toquickly and efficiently conduct a transaction such as selling, buying,or returning a good. The electronic device may use near fieldcommunications technology to perform these and other functions.

As mentioned above, a band may be connected to two electronic devicesand may serve as a wired communication path between the two. As anotherexample, the devices may communicate wirelessly, thereby permitting onedevice to relay information from a second to a user. This latter examplemay be particularly useful when the second is inaccessible.

Certain embodiments may incorporate one or more biometric sensors tomeasure certain physiological characteristics of a user. The device mayinclude a photoplesymogram sensor to determine a user's heart rate orblood oxygenation levels, for example. The device may also or insteadinclude electrodes to measure the body impedance of a user, which maypermit the device to estimate body fat percentages, the body'selectrical activity, body impedance, and so on. Also include bloodpressure, ultraviolet exposure, etc. Depending on the sensorsincorporated into or associated with the electronic device, a variety ofuser characteristics may be measured and/or estimated, therebypermitting different health information to be provided to a user.

Certain embodiments may be wirelessly charged. For example, an inductivecharging base may transmit power to an inductive receiver within thedevice in order to charge a battery of the device. Further, by varyingthe inductive field between the device and base, data may becommunicated between the two. As one simple non-limiting example, thismay be used to wake the base from a low-power sleep state to an activecharging state when the device is placed on the base. Other wirelesscharging systems also may be used (e.g., near field magnetic resonanceand radio frequency). Alternatively, the device also may employ wiredcharging through electrodes.

In certain embodiments, the device may include a rotary input, which maytake the form of a crown with a stem. The crown and stem may be rotatedto provide the rotary input. Rotation of the stem and/or crown may besensed optically, electrically, magnetically, or mechanically. Further,in some embodiments the crown and stem may also move laterally, therebyproviding a second type of input to the device.

The electronic device may likewise include one or more buttons. Thebutton(s) may be depressed to provide yet another input to the device.In various embodiments, the button may be a dome switch, rocker switch,electrical contact, magnetic switch, and so on. In some embodiments thebutton may be waterproof or otherwise sealed against the environment.

Various embodiments may include or otherwise incorporate one or moremotion sensors. A motion sensor may detect motion of the device andprovide, modify, cease, or otherwise affect a state, output, or input ofthe device or associated applications based on the motion. Asnon-limiting examples, a motion may be used to silence the device oracknowledge an alert generated by the device. Sample motion sensorsinclude accelerometers, gyroscopic sensors, magnetometers, GPS sensors,distance sensors, and so on. Some embodiments may use a GPS sensor tofacilitate or enable location and/or navigation assistance.

As shown in FIG. 24, the device 100 may also include one or moreacoustic elements, including a speaker 2440 and/or a microphone 2445.The speaker 2440 may include drive electronics or circuitry and may beconfigured to produce an audible sound or acoustic signal in response toa command or input. Similarly, the microphone 2445 may also includedrive electronics or circuitry and is configured to receive an audiblesound or acoustic signal in response to a command or input. The speaker2440 and the microphone 2445 may be acoustically coupled to port oropening in the case that allows acoustic energy to pass, but may preventthe ingress of liquid and other debris.

Certain embodiments may incorporate an ambient light sensor. The ambientlight sensor may permit the device to sense a brightness of itsenvironment and adjust certain operational parameters accordingly. Forexample, the electronic device may modify a brightness of a display inresponse to the sensed ambient light. As another example, the electronicdevice may turn the display off if little or no light is sensed for aperiod of time.

These and other functions, operations, and abilities of the electronicdevice will be apparent upon reading the specification in its entirety.

Alert Class Based Haptic Output

Alert conditions can be triggered in many different ways according tosome embodiments. For example, some alerts are triggered by acommunication from another person, such as receiving an email, receivinga voicemail message, and receiving a text message. Other alerts aretriggered by user-set notifications, such as a calendar alert orreminder. Yet other alerts are triggered by system based notifications,such as application-based or device-based alerts, such as a financialtransaction event or a device pairing event. Some alerts are triggeredby third party applications, whereas other alerts are triggered fromapplications or other aspects native to the device. What type oftriggered resulted in the alert provides the basis for providing acorresponding haptic output, according to some embodiments.

A corresponding haptic is associated with an alert class according tosome embodiments, allowing the user to, over time, come to understandthe haptic feedback as a language with specific haptic and other outputassociated with specific alert classes. Providing haptic output thatcorrelates with various alert classes creates a more efficienthuman-machine interface, thereby reducing the amount of time it takesfor a user to understand alerts and perform operations, whichconsequently reduces energy use and increases battery life for batterypowered devices. FIG. 25 is a flow diagram of a method 2500 forproviding various haptic and other feedback alerts, each correspondingto a different alert class according to some embodiments. Note that insome embodiments, different steps may be performed other than thoseshown in FIG. 25.

The method begins by the device 100 detecting 2505 an event. For anapplication executing on a computing device 100 with a tactile outputgenerator, the event takes various forms according to variousembodiments. An event is any notification, alert, or other eventdirected to a user of the device. In some circumstances, the device 100has a touch-sensitive display.

Different types of alert events correspond to different event classesaccording to some embodiments. In accordance with a determination of theevent type falling into a particular class, an alert is providedcorresponding to the event class. For example, in accordance with adetermination that the event is a type of event in the first eventclass, a first alert is provided 2515 that includes a first hapticcomponent generated by the tactile output generator. Similarly, inaccordance with a determination that the event is a type of event in asecond event class that includes a plurality of different event types, asecond alert is provided 2520 that includes a second haptic component,generated by the tactile output generator, that is different from thefirst haptic component. Likewise, in accordance with a determinationthat the event is a type of event in a third event class that includes aplurality of different event types, a third alert is provided 2525 thatincludes a third haptic component, generated by the tactile outputgenerator, that is different from the first haptic component and thesecond haptic component. Thus, the haptic output depends on the eventclass that the event belongs to according to some embodiments.

According to some embodiments, the first haptic component is differentfrom the second haptic component and the third haptic component based onone or more of spacing between haptic outputs, number of haptic outputs,and change in the amplitude of haptic outputs over time during thehaptic component.

In some embodiments, the haptic components are designed to besubstantially different so as to enable a user to easily distinguishbetween the different classes of alerts whether or not an audiblecomponent of the alert is provided by the device, as discussed inconjunction with FIG. 25 above. For example, the device may have aplurality of unique alerts that correspond to different events, and thefirst event class, the second event class, and the third event classcorrespond to types of events that typically occur more frequently thanthe events that correspond to unique alerts.

In some circumstances the event classes corresponds to events generatedbased on different factors. For example, according to some embodiments,a first class of events includes events generated based oncommunications from other users, a second class of events includesevents generated based on predefined criteria entered by a user of thedevice, and a third class of events includes events generated based onthe satisfaction of device-monitored criteria.

According to some embodiments, the event classes correspond to variousevents, as further discussed below in conjunction with Table 9. Forexample, the first type of event is in a personal event class, thesecond type of event is in a user-set event class, and the third type ofevent is in a system event class, according to some embodiments. Apersonal event class may include events such as incoming messages,incoming email, incoming voicemail, etc. A user-set event class mayinclude events such as calendar notifications, reminder notifications,timer/alarm events, etc. A system event class may include events such asa pairing event, a financial transaction event, or a battery levelevent, etc. According to some embodiments, the alerts corresponding toeach class are pre-configured for that class, such that the class sharesone or more attribute of the haptic output. According to someembodiments, the haptic corresponding to a class is configurable by theuser of the device 100.

According to some embodiments, in response to detecting 2506 the event,and in accordance with a determination that the event is a type of eventin a fourth event class that includes a plurality of different eventtypes, providing 2530 a fourth alert that includes a fourth hapticcomponent, generated by the tactile output generator, that is differentfrom the first haptic component, the second haptic component, and thethird haptic component. The fourth event class corresponds to an eventgenerated by a third party application according to come embodiments,which may include various second-party or third party events, forexample a social network providing a social networking message or a gameapplication providing a game notification.

According to some embodiments, the fourth haptic component isdifferentiated from the first haptic component, the second hapticcomponent, and the third haptic component by a haptic output elementthat is not included in the first haptic component, the second hapticcomponent, or the third haptic component. For example, in the case ofthe fourth event class corresponding to an event generated by a thirdparty application, the fourth haptic component may include a hapticcomponent unique to third party events (e.g., a particular vibrationpattern) and not included in the haptic output provided corresponding tothe other classes, such that a user can easily tell that an alert isfrom a third party application rather than from any of the nativeapplications or features that are supplied by the device. In thisexample, the first, second, and third event classes correspond to eventsinitiating from the device directly or from an application native to thedevice. However, in some embodiments a haptic component corresponding toa given class may be unique to that class and not included in the hapticoutput provided corresponding to the other classes, such that a user caneasily tell from the haptic output that an alert is from a particularevent class as distinguished from other classes.

In some embodiments, if the haptic output is accompanied by respectiveaudio component output, the audio components are reflective of the eventclass as well. For example, the audio component for a class may beunique to that class (e.g., a particular type of sound) and not includedin the audio output provided corresponding to the other classes, suchthat a user can easily tell from the audio output that an alert is froma particular event class as distinguished from other classes.

Table 9 shows a few exemplary circumstances under which the device 100has information used to determine that the detected event is of a knownevent class. The examples shown represent four event classes: personal,user-set, system, and third party, but could be any event class. Theevent types shown in Table 9 include email messages, text messages, andvoicemail messages (Personal), calendar and reminder notifications,timer and alarm alerts (User-Set), device pairing, financialtranslations, and battery level indications (System), and third partymessages and notifications (Third Party), but could be any other eventsproviding similar information to the device 100 for making thisdetermination.

TABLE 9 MULTIPLE EVENT CLASS HAPTICS Event Haptic Audio Class Event TypeCondition Output Output Personal Email Email Fast double tap /Ding longmessage received fade Personal Text Text message Fast double tap /Dinglong message received fade Personal Voicemail Voicemail Fast double tap/Ding long received fade User-Set Calendar Calendar Slow double /Chimenotification tap User-Set Reminder Reminder Slow double /Chimenotification tap User-Set Timer Timer end Slow double /Chime tapUser-Set Alarm Alarm time Slow double /Chime reached tap System DevicePairing event Rumble Dit (o)/Ding pairing short fade System FinancialTransaction Rumble Dit (o)/Ding application event short fade SystemBattery level Low battery Rumble Dit (o)/Ding short fade Third Thirdparty Message Fast double /Ding up (o) Party message received tap, buzzfade Third Third party Notification Fast double /Ding down Partynotification event tap, buzz fade (o)

For the examples shown, the event class column shows which of fourclasses the event falls into: Personal, User-Set, System, or ThirdParty. Within each class, various event types are listed per the eventtype column. A few examples are listed for each event class, but thelist is not meant to be exclusive of other event types. In someembodiments, Personal events are those sent to the device userpersonally, typically by another user, such as an email, text, orvoicemail. User-Set events are events that, as the name implies, are setby the user of the device, e.g., calendar notifications, reminders,timer expiration, alarms going off, etc., according to some embodiments.In some embodiments, System events are events generated by the device orsystem itself, such as a device pairing or batter level indication.Third Party events can vary greatly, but generally correspond to anevent triggered by a third party application running on the device,according to some embodiments. Examples of third party events includemessages or notifications corresponding to a social networkingapplication, notifications or reminders corresponding to a gamingapplication, activity alerts corresponding to a fitness application, orthe like.

Note that in the examples shown, regardless of the event type, everyevent corresponding to a given event class has the same haptic and audiooutput. According to some embodiments, all Personal event class eventsresult in output of a haptic component that includes a fast double tapfirst, followed by a ding audio component with a long fade. In thisexample, the output is perceived by the user as tap-tap-ding (fade).According to some embodiments, all User-Set event class events result inoutput of a haptic component that includes a slow double tap first,followed by a chime audio component. In this example, the output isperceived by the user as tap-tap-chime.

According to some embodiments, all System event class events result inoutput of a haptic rumble that overlaps at the end of the rumble with aquieter “dit” audio component, followed by a ding audio component with ashort fade that begins after completion of the rumble. In this example,the output is perceived by the user as rumble . . . (dit) . . . -ding(fade). According to some embodiments, all Third Party event classevents result in output of a consistent haptic component correspondingto the third party applications including a fast double tap and a buzzfade, with the buzz fade over lapping in part with an audio component,which may vary by the event type or condition associated with the thirdparty application. In this example, the output is perceived by the useras tap-tap-ding (up or down) . . . (buzz (fade)) . . . .

Note that in these examples, both the haptic and audio componentsuniquely correspond to a given class, such that a user can easily tellfrom the output that an alert is from a particular event class asdistinguished from other classes. In the example of the third partyapplication class, the haptic is identical and the audio ischaracteristic of the class but varies depending on the event type orcondition. In addition, the haptic output portion alone also uniquelycorresponds to a given class in these examples, such that a user cantell from the haptic output alone that an alert is from a particularevent class, even if the user has the sound turned off on his or herdevice. In order to provide users with a consistent and easilyunderstandable alert language, the different classes of alerts eachcorrespond to multiple different events, and in embodiments where thealerts are user selectable (or user modifiable), users are optionallyprovided with options for changing a sound or haptic output associatedwith an event class, but changing that sound or haptic output will causethe sound or haptic output to change for all events that are associatedwith that class of alerts. Additionally, in some embodiments, users areprovided with the option to assign events to different event classes.This customizability enables users to customize the alerts provided bytheir device while still maintaining a consistent alert language thatenhances the efficiency and effectiveness of the alerts.

It should be understood that the particular order in which theoperations in FIG. 25 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1200, 1300, 2600, 3000, and 3200) are alsoapplicable in an analogous manner to method 2500 described above withrespect to FIG. 25. For example, the inputs, alert conditions, events,applications, states, and haptic outputs described above with referenceto method 2500 optionally have one or more of the characteristics of theinputs, alert conditions, events, applications, states, and hapticoutputs described herein with reference to other methods describedherein (e.g., methods 700, 900, 1000, 1100, 1200, 1300, 2600, 3000, and3200). For brevity, these details are not repeated here.

Selective Alert Salience for Haptic Output

Alert conditions are selectively subject to an increase in the salienceof the haptic output according to some embodiments. In some embodiments,the device 100 can modify the salience of the haptic output to “prime”the user, which helps draw greater attention to the alert. For example,the device 100 can provide a priming haptic output as a precursor toanother haptic output, the priming haptic output designed to increasethe attentiveness of a user operating the device 100 to the other hapticoutput. In some circumstances, the salience is increased for somehaptics and not others. This is particularly important for users withsensory impairments (e.g., users who cannot hear audible alerts or havereduced sensitivity to vibrating alerts).

FIG. 26 is a flow diagram of a method 2600 for providing selectivesalience increase for various haptics, according to some embodiments.Note that in some embodiments, different steps may be performed otherthan those shown in FIG. 26.

The method 2600 begins by detecting 2605 the occurrence of a first eventat a computing device with a tactile output generator. The event can beof any type, including the event types discussed in conjunction withFIG. 25 above. In response to detecting the occurrence of the firstevent, events of different types may produce different alerts withdifferent salience. For example, in accordance with a determination thatthe first event is a first type of event, a first alert is provided 2610with the tactile output generator that includes a first hapticcomponent, and the haptic component is selected based at least in parton an alert-salience setting of the device. In accordance with adetermination that the first event is a second type of event, a secondalert is provided 2630, with the tactile output generator that includesa second haptic component selected without regard to the alert-saliencesetting of the device.

According to some embodiments, the alert-salience setting of the devicehas two values, with one value corresponding to increasing salience ofthe alert and the second value corresponding to not increasing thesalience of the alert. FIG. 27 illustrates an exemplary user interfacefor an alert-salience setting on a device 100 in accordance with someembodiments. As shown, an increased salience alert selector 2710 allowsthe user to turn use of increased salience on or off, such that “on”indicates that certain alerts will have increased salience, and “off”indicates that no alerts will have increased salience. As discussedherein, some alerts will have increased salience when this setting ison, and some will not take into account the setting. For example, usingthe exemplary classes discussed in conjunction with FIG. 25, when thealert-salience setting is on, events in the Personal and User-Set eventclasses will result in haptic output with increased salience, and eventsin the System and Third Party event classes will be unchanged despitethe alert-salience setting being on, according to some embodiments.

When an event type is part of a binary set of related events, one of thetwo events can be designated as receiving an increased salience hapticaccording to some embodiments. For example, buddy pairing, whenattempted has only two results: successful buddy pairing or failed buddypairing. For successful budding pairing, increased salience may not benecessary as it is the expected outcome. However, a failure to pair maybe unexpected, and thus can be called to the user's attention via anincreased saliency haptic to more easily gain the user's attention.Thus, for the buddy pairing event failure result of the binary options,an increased saliency haptic would result, whereas for a buddy pairingsuccess a standard haptic output would be provided, according to someembodiments.

In some embodiments, whether or not an alert has increased salience whenthe increased-salience setting is on is based on an importance of theevent that triggered the alert. For example, in some embodiments eventscan include information (e.g., in the form of metadata) corresponding tothe importance or urgency of the event, such as an urgent or importantflag on an email, a message received from a VIP contact, or othersimilar information indicating that the event is in some way moreimportant than other events. In some embodiments, the content of theevent may be examined for keywords indicating importance such as“urgent,” “high importance,” etc. In some embodiments, an even higherdegree of salience can be added for events with more than one increasedsalience indicator. For example, when a message is received and theincreased salience is off, the message triggers a standard haptic forthe Personal event class. However, when the same message is receivedfrom a VIP contact and the salience setting is off, some increase insalience still may be applied to distinguish the VIP contact over othermessaging contacts. And when the increased salience is on, messagesreceived increased salience. Thus, according to some embodiments, anincrease from one trigger (e.g., increased salience setting forPersonal) can be combined with an increase from another trigger (e.g.,message is from a VIP contact) to produce an “extra salience” output.For example, the extra salience output could be three separate priminghaptic components preceding the standard haptic for the event class.

Some event types are deemed more important by way of the nature of theevent, according to some embodiments. For example, events triggeredfinancial transactions in some instances are considered more importantthan other events, regardless of the event class that the financialtransactions fall within. Thus, specific categories of events consideredmore sensitive—in the example of financial transactions, these couldinclude any activity associated with banking or stock tradingapplications, or other events related to money and/or more subject tofraud activities. Another example is events within an event classsubject to increased salience (e.g., Personal) that also are messageevent types likely to have been received directly from a human userother than the user operating the device. For example, an incomingmessage or call from a known email address or phone number on the user'scontact list. For further discussion of event types likely to have beenreceived from a human user other than the user operating the device, seeTable 2. In this example, all message event types likely to have beenreceived from a human user have an increased salience haptic (e.g.,mail, messages, and voicemail), but other event types in the class donot (e.g., photostream activity).

According to some embodiments, certain event types may be excluded fromincreased haptic output salience, even when other event types in theevent class provide the increased salience output corresponding to anincreased salience setting. For example, within an event class (e.g.,Personal) with increased saliency on, events triggered by responses toUI interactions (e.g., scrolling, navigating through items, pressing UIbuttons, or other user input) do not have increased salience alerts evenwith the setting on. In some instances, the user input is characteristicof a high level of user engagement with the application or device, andthus is an event that doesn't require increased salience. Thus, the useris not bombarded with an unduly strong feedback signal when the user hasa high degree of engagement or interaction with the application/device.

Another condition under which certain event types may be excluded fromincreased haptic output salience, even when other event types in theevent class provide the increased salience output corresponding to anincreased salience setting, is when the application providing the alertis in an active state (e.g., being used by the device user). Forexample, a received message event type is in the Personal event class,and thus typically would be subject to increased salience alerts whenthe setting is on. However, in the event that the user of the device isusing the messaging application at the time the message is received, anincreased salience haptic would not be provided. On the other hand, thesame received message event would trigger increased salience if the userwas not using the messaging application when the message was received.

In the example above in which a first event type results in an alertthat includes a haptic component selected based at least on thealert-salience setting and a second event type selects a hapticcomponent without regard to the alert-salience setting, when the settingvalue corresponds to increasing salience of the alert, the first alertwould have increased salience and the second alert would not. Thus,despite the alert-salience setting having a value corresponding toincreasing salience of the alert, not all alerts would have increasedsalience. Continuing the example from above, if the first type of eventis an incoming message event, corresponding to a Personal event classtype of event, and the second type of event is a payment event,corresponding to a System event class type of event, and the settingvalue corresponds to increasing salience of the alert, the message(Personal class) event would have increased salience but the payment(System class) event would not, since the second type does not take thesetting into consideration. According to some embodiments, Personal andUser-Set event classes would have increased salience haptics for alertswhen the alert-salience setting is on, whereas System and Third Partyevent classes would have the standard haptic alert regardless of thealert-salience setting value.

According to some embodiments, the first alert is different from thesecond alert. For example, the first alert could include a differenthaptic component than the second alert, or the first alert could includea different audio component than the second alert, or the first alertcould include both a different haptic component and a different audiocomponent.

However, in some embodiments the device forgoes the additional hapticcomponent and just provides 2625 the standard haptic component foralerts triggered in response to events in the first plurality of eventclasses when the alert-salience setting has the second value. Continuingwith the above examples related to classes, a standard haptic componentwould result from alerts triggered in response to Personal or User-Setevent class events, when the alert-salience setting has the second value(off). Likewise, in some embodiments the device forgoes providing theadditional haptic component and just provides 2625 the standard hapticcomponent for alerts triggered in response to events in a secondplurality of event classes, without regard to the alert-salience settinghas the first value or the second value. Continuing with the aboveexamples related to classes, a standard haptic component would alsoresult from alerts triggered in response to System or Third Party eventclass events, regardless of the alert-salience setting value. In somecases the second plurality of event classes is distinct from the firstplurality of event classes. Thus, in some examples the additional hapticcomponent is provided only if an incoming event is both of an eventclass for which the device increases alert-salience and thealert-salience value is set to increasing the salience of the alert. Forexample, an event in the Person or User-Set event classes, when thealert-salience setting is “on.”

According to some embodiments, the method for providing selectivesalience increase for various haptics includes first determining 2615 avalue of the alert-salience setting. In accordance with a determinationthat the alert-salience setting has a first value, the first alertincludes 2620 a first haptic component and a second haptic component,while in accordance with a determination that the alert-salience settinghas a second value different from the first value, the first alertincludes 2625 the first haptic component but does not include the secondhaptic component. As an example, the first value could be an “increasealert salience” value and the second value could be a “do not increasealert salience” value. Continuing with the above examples, for an eventin the Personal or User-Set event classes, and an alert-salience settingof “increase alert salience” (on), according to some embodiments alertswould include both the first haptic component—the standard hapticcomponent for the corresponding event class—and a second hapticcomponent to increase the saliency of the haptic output associated withthe alert. For an event in the System or Third Party event classes, evenwith an alert-salience setting of “increase alert salience” (on),according to some embodiments alerts would include only the first,standard haptic component for the event corresponding class. Accordingto some embodiments, the method further includes in accordance with adetermination that the first event is a third type of event, providing athird alert that includes a haptic component selected based at least inpart on the alert-salience setting of the device. In this example,providing the third alert that includes the haptic component selectedbased at least in part on the alert-salience setting of the deviceincludes determining a value of the alert-salience setting (e.g.,similar to step 2615 above). Referring again to the event classexamples, if the Personal event class was the first type, the User-Setevent class is the third type of this example, according to someembodiments.

According to some embodiments, in accordance with a determination thatthe alert-salience setting has the first value, the third alert includesa third haptic component and a fourth haptic component, and inaccordance with a determination that the alert-salience setting has thesecond value different from the first value, the third alert includesthe third haptic component but does not include the fourth hapticcomponent. As with the prior example, the first value is an increasealert salience value and the second value is a do not increase saliencevalue, according to some embodiments. For example, the third hapticcomponent—the standard haptic component for the corresponding eventclass—would be paired with the fourth haptic component, to increase thesaliency of the haptic output associated with the alert. In addition, insome embodiments, the fourth haptic component is the same as the secondhaptic component. For example, both the first haptic component and thesecond haptic component are increased-salience haptic, or priming,components that are used to notify the user that an alert is imminent.In some embodiments, the device provides the second haptic outputfollowing and within a specified time interval of providing the priminghaptic output. Continuing with the event class examples, the fourthhaptic, corresponding to increased salience of event alerts for eventsin the User-Set event class, is the same as the second haptic,corresponding to increased salience of event alerts for events in thePerson event class, both of which haptic components are a priming hapticthat precedes the standard haptic for each class.

According to some embodiments, the method further includes, when thefirst alert is detected while the alert-salience setting of the devicehas the first value, that the first alert provided in response to thefirst event includes the first haptic component and the second hapticcomponent and the second alert provided in response to the first eventincludes a second-alert haptic component. The method goes on, in someembodiments, to receive a request to change the alert-salience settingof the device to the second value while the alert-salience setting ofthe device has the first value, and in response to receiving the requestto change the alert-salience setting, changing the alert-saliencesetting to the second value. For example, if for Personal event classevents the first and second haptic components have been provided due tothe alert-salience setting being on, the request is a request to turnthe alert-salience setting off. Similarly, the method includes in someembodiments, while the alert-salience setting of the device has thesecond value, detecting occurrence of a second event, and in responseproviding a third alert in accordance with a determination that thesecond event is the first type of event, the third alert including thefirst haptic component and not including the second haptic component.For example, the third haptic component only has been provided forUser-Set event class events, because the alert-salience setting isturned off. Likewise, in accordance with a determination that the secondevent is the second type of event, the method proceeds with providingthe second alert that includes the second-alert haptic component. Insome embodiments, the second alert does not include the second hapticcomponent (e.g., the second alert is not affected by the alert-saliencesetting and does not include a the second “extra salience” hapticcomponent whether the alert-salience setting is set to “increase alertsalience” “do not increase alert salience”). For example, the secondtype of event is a System event class event, and the second hapticcomponent is not included in the output regardless of the alert-saliencesetting.

According to some embodiments, the first haptic component is differentfrom the second haptic component and the third haptic component based onone or more of spacing between haptic outputs, number of haptic outputs,and change in the amplitude of haptic outputs over time during thehaptic component.

In some embodiments, the haptic components are designed to besubstantially different so as to enable a user to easily distinguishbetween the different classes of alerts whether or not an audiblecomponent of the alert is provided by the device, as discussed inconjunction with FIG. 25 above. For example, the device may have aplurality of unique alerts that correspond to different events, and thefirst event class, the second event class, and the third event classcorrespond to types of events that typically occur more frequently thanthe events that correspond to unique alerts. As it relates to classes, ahaptic component of an alert for an event in the first event class isdistinguishable by the user from a haptic component of an alert for anevent in the second event class, both of which are distinguishable bythe user from a haptic component of an alert for an event in the thirdevent class. In some cases, events that typically occur more frequentlyare events that, statistically speaking, occur more frequently thanother events. As discussed above in conjunction with Table 9, in someembodiments, events that occur more frequently are grouped together intoevent classes and provided with a consistent alert so that a user canlearn to differentiate between the different types of events that occurfrequently.

In some embodiments, the event class associated with the event impactswhether an additional haptic component is provided in conjunction with atriggered alert. According to some embodiments, an additional hapticcomponent is provided 2620 for alerts triggered in response to events ina first plurality of event classes (e.g., Personal) when thealert-salience setting has a first value. For example, in addition to ahaptic associated with the first event class, an additional haptic isprovided as well. The additional haptic is a priming haptic in someembodiments. For example, in some circumstances the second hapticcomponent precedes the first haptic component.

In some circumstances the event classes corresponds to events generatedbased on different factors. For example, according to some embodiments,a first class of events includes events generated based oncommunications from other users, a second class of events includesevents generated based on predefined criteria entered by a user of thedevice, and a third class of events includes events generated based onthe satisfaction of device-monitored criteria. According to someembodiments, the event classes are characterized as belonging to a classby the event information, similar to the event classes discussed inconjunction with FIG. 25: Personal, User-Set, System, and Third Party.

In some embodiments, the event classes are configurable such that thedevice enables a user to select which types of events fall into an eventclass. However, in some embodiments, the types of events that fall intoan event class are predefined, and the device does not enable a user ofthe device to select which event types fall into a particular eventclass.

Similarly, according to some embodiments, the alert-salience setting isuser configurable, such that the user can select specific event classesor even individual event types for which the user would like increasedsaliency in the haptic output. Allowing the user to customize hapticoutput that correlates with events the user deems more important createsa more efficient human-machine interface, thereby reducing the amount oftime it takes for a user to perform operations, which consequentlyreduces energy use and increases battery life for battery powereddevices.

Table 10 shows a few exemplary circumstances under which the device 100has information to selectively increase salience based on analert-salience setting. The examples shown represent the same four eventclasses as shown above in Table 9: Personal, User-Set, System, and ThirdParty, but could be any event class.

TABLE 10 SELECTIVE HAPTIC SALIENCE Event Salience Haptic Audio ClassEvent Type Condition Setting Output Output Personal Email Email receivedOff Fast double tap /Ding long message fade Personal Email Emailreceived On Salient buzz; /Ding long message Fast double tap fadePersonal Text Text message Off Fast double tap /Ding long messagereceived fade Personal Text Text message On Salient buzz; /Ding longmessage received Fast double tap fade User-Set Calendar Calendar OffSlow double /Chime notification tap User-Set Calendar Calendar OnSalient buzz; /Chime notification Slow double tap User-Set ReminderReminder Off Slow double /Chime notification tap User-Set ReminderReminder On Salient buzz; /Chime notification Slow double tap SystemDevice Pairing event Off Rumble Dit (o)/Ding pairing short fade SystemDevice Pairing event On Rumble Dit (o)/Ding pairing short fade SystemFinancial Transaction event Off Rumble Dit (o)/Ding application shortfade System Financial Transaction event On Rumble Dit (o)/Dingapplication short fade Third Third party Message received Off Fastdouble /Ding up (o) Party message tap, buzz fade Third Third partyMessage received On Fast double /Ding up (o) Party message tap, buzzfade Third Third party Notification event Off Fast double /Ding downParty notification tap, buzz fade (o) Third Third party Notificationevent On Fast double /Ding down Party notification tap, buzz fade (o)

For the examples shown, the event class column shows which of fourclasses the event falls into: Personal, User-Set, System, or ThirdParty. Within each class, various event types are listed per the eventtype column. A few examples are listed for each event class, but thelist is not meant to be exclusive of other event types. As with theprevious chart, regardless of the event type, every event correspondingto a given event class has the same standard haptic and an audio outputcharacteristic of the class, and each class shares the same increasedsalience output as well for any events corresponding to the event class.In the example of the third party application class, the haptic isidentical and the accompanying audio component is characteristic ofthird party applications even though the audio varies by event type orcondition.

Table 10 includes an extra column corresponding to the alert-saliencesetting. Note that when the alert-salience setting is off, indicatingthat increased saliency should not be used, the standard haptic andaudio output is shown, which corresponds to the examples in Table 9.This maintains the goal of both the haptic and audio components uniquelycorresponding to a given class, such that a user can easily tell fromthe output that an alert is from a particular event class asdistinguished from other classes.

For two of the event classes, Personal and User-Set, when thealert-salience is turned on, indicating that increased saliency shouldbe used, a salient buzz haptic is added to the standard output for theclass. According to some embodiments, the additional haptic is added tothe beginning of the alert such that it precedes the standard output. Inthis example, a salient buzz haptic component is added, which is aseries of taps of a longer duration and of high frequency, such that theuser feels a constant vibration or “buzz.” According to someembodiments, the salient buzz has a duration of 0.5 seconds or more, andramps up in intensity from the beginning of the buzz to the end, whichassists in getting the user's attention prior to the standard output.Also note that even when the alert-salience setting is on, two of theevent classes, System and Third Party, show no increase in saliency ofthe output, i.e., the standard output is used regardless of the setting.

TABLE 11 ADDITIONAL ALERT TYPES AND CLASSES Haptic Haptic (Increased(Increased Salience = Salience = Type Class Event Audio Off) On)Notification 3rd Party Notification 3rd Party Alert A004 H004 H066Notification Personal Notification Mail A001 H001 H066 NotificationPersonal Notification Messages (App A001 H001 H066 Closed) NotificationPersonal Notification Photostream A001 H001 H001 Activity NotificationPersonal Notification Voicemail A001 H001 H066 Received NotificationSystem Notification Payment Geofence A002 H002 H002 Notification SystemNotification Payment Balance A002 H002 H002 Notification SystemNotification Device Pairing A002 H002 H002 Success Notification SystemNotification Device Pairing A002 H002 H002 Connection NotificationUser-Set Notification Calendar A003 H003 H067 Notification User-SetNotification ToDo List A003 H003 H067 Reminder Feedback Camera CameraA010 H010 H010 Countdown Start Feedback Camera Camera A011 H011 H011Countdown End Feedback Camera Camera Shutter A012 H012 H012 FeedbackCommunication Messages (App A022 H022 H022 open, incoming) FeedbackCommunication Messages (App A023 H023 H023 open, outgoing) FeedbackCommunication ET Send A019 H019 H019 Feedback Communication Heartbeat(A020) H020 H020 Feedback Communication Walkie Talkie A033 H033 H033Feedback Communication Walkie Talkie A034 H034 H034 Start FeedbackCommunication Walkie Talkie End A035 H035 H035 Feedback CommunicationCall End A057 H057 H057 Feedback Communication Electronic Touch A060H060 H060 Receive Feedback Communication Electronic Touch A065 H065 H068Begin Feedback Communication Pairing Failure A021 H021 H021 FeedbackCommunication Device Pairing A057 H057 H057 Remote Tap Feedback PaymentPayment Failure A021 H021 H021 Feedback Payment Payment Success A032H032 H032 Feedback Phone Phone Answer A008 H008 H008 Feedback PhonePhone Hold A008 H008 H008 Feedback Phone Ringtone A028 H028 H028Feedback UI Controls Stopwatch Reset A008 H008 H008 Feedback UI ControlsStopwatch Start A008 H008 H008 Feedback UI Controls Stopwatch Stop A008H008 H008 Feedback UI Controls Stopwatch Lap A008 H008 H008 Feedback UIControls Timer Cancel A008 H008 H008 Feedback UI Controls Timer StartA008 H008 H008 Feedback UI Controls Timer Pause A008 H008 H008 FeedbackUI Controls Workout Select A008 H008 H008 Feedback UI Controls WorkoutStart A008 H008 H008 Feedback UI Controls Timer Wheel Hour A016 H016H016 Detent Feedback UI Controls Timer Wheel A016 H016 H016 MinuteDetent Feedback UI Controls Detent A017 H017 H017 Feedback UI ControlsZoom Detent A017 H017 H017 Feedback UI Controls End of List (A036) H036H036 Feedback Feedback UI Controls UI Edge Reached A036 H036 H036Feedback UI Controls UI Turn Switch A037 H037 H037 On Feedback UIControls UI Turn Switch A037 H037 H037 Off Feedback UI Controls UI SwipeA038 H038 H038 Feedback UI Controls Force Touch (A039) H039 H039 InvokeFeedback UI Controls Force Touch Exit A040 H040 H040 Feedback UIControls Silence A041 H041 H041 Notifications Feedback UI Controls UIConfirmation A048 H048 H048 Feedback UI Controls UI Snap A048 H048 H048Feedback UI Controls Preview A050 H050 H050 Feedback Virtual AssistantVirtual Assistant (A029) H029 H029 Start Feedback Virtual AssistantVirtual Assistant (A030) H030 H030 Success Feedback Virtual AssistantVirtual Assistant (A031) H031 H031 Failure Alert Activity SedentaryTimer A005 H005 H005 Alert Activity Activity Goal A006 H006 H006 BehindAlert Activity Activity Goal A006 H006 H006 Close Alert ActivityActivity Goal A007 H007 H007 Attained Alert Activity Workout CompleteA007 H007 H007 Alert Activity Workout A009 H009 H009 Countdown AlertNavigation Navigation A024 H024 H024 Generic Alert NavigationNavigation: Left A025 H025 H025 Turn Alert Navigation Navigation: RightA026 H026 H026 Turn Alert Stopwatch/Timer/Clock Clock Tick A013 H013H013 Alert Stopwatch/Timer/Clock Hourly Chime A014 H014 H014 AlertStopwatch/Timer/Clock Alarm A015 H015 H015 Alert Stopwatch/Timer/ClockTimer A015 H015 H015 Alert System Alerts End of Do Not A042 H042 H042Disturb Alert System Alerts System Startup A043 H043 H043 Alert SystemAlerts Unlock A044 H044 H044 Companion Device Alert System AlertsCharging A045 H045 H045 Alert System Alerts Unlock Failure A055 H055H055 Alert System Alerts Unlock Success A056 H056 H056 Alert SystemAlerts Dictation Start A058 H058 H058 Alert System Alerts Dictation EndA059 H059 H059 Alert System Alerts Battery Low A061 H061 H061 AlertSystem Alerts Phone Failure A069 H069 H069

In Table 11, shown above, the different audio outputs and haptic outputsthat correspond to different events are shown. The events are dividedinto Notifications, Feedback, and Alerts. Audio outputs with the samename correspond to the same (or substantially the same) audio output(audio waveforms). Audio outputs with different names correspond todifferent (or materially/perceptibly different) audio outputs (audiowaveforms). Haptic outputs with the same name correspond to the same (orsubstantially the same) haptic output (haptic waveforms). Haptic outputswith different names correspond to different (or materially/perceptiblydifferent) haptic outputs (haptic waveforms). Haptic/Audio output namesthat are enclosed in brackets ( ) are, optionally omitted and replacedwith the absence of an audio output or haptic output.

In some embodiments, haptic/audio outputs are grouped into a pluralityof types and or classes in addition to the notification classesdescribed above. Notifications generally correspond to haptic/audiooutputs that are designed to get the user's attention when the user isnot paying attention to the device (e.g., when a new message is receivedor it is time for a calendar event to occur). Alerts generallycorrespond to haptic/audio outputs that are designed to alert the userto operations being performed at the device (or conditions of thedevice) while the user is paying attention to the device (e.g.,navigation instructions while the user is walking or driving, deviceunlock failure/success, etc.). Feedback generally corresponds tohaptic/audio outputs that are designed to give the user feedback thatinputs from the user have been detected by the device (e.g., scrollinginputs, button selection inputs, sent messages, etc.), or to indicatethat the input has caused the device to reach some threshold (e.g., ascrolling limit, a zoom limit, a force/pressure threshold at which analternative menu is displayed). In some embodiments, when the increasedsalience setting is “on,” increased salience haptic/audio output isprovided for notifications, which are designed to get the user'sattention when the user is not paying attention, but not for feedback(e.g., because the user is already paying attention to and activelyinteracting with the device). In some embodiments, increased saliencehaptic/audio output is provided for some notifications associated withan application (e.g., message notifications) when the application isclosed but not for feedback associated with the application when theapplication is open (e.g., sent and received message alerts when amessaging application is open). For example in Table 11, a MessagesNotification uses different haptic outputs based on the increasedsalience setting (e.g., haptic output H001 when the increased saliencesetting is off and haptic output H066 when the increased saliencesetting is on) but Messages Feedback when a message is sent uses thesame haptic output without regard to the increased salience setting(e.g., haptic output H023 is used if the increased salience setting ison or off).

In some embodiments the alert salience setting changes a salience ofhaptic output without changing a salience of corresponding audio output.For example, as shown above in Table 11, the Messages notification usesthe same audio output (A001) whether or not the increased saliencesetting is on, but uses a different haptic output depending on theincreased salience setting (e.g., haptic output H001 is used when theincreased salience setting is off and haptic output H066 is used whenthe increased salience setting is on).

It should be understood that the particular order in which theoperations in FIG. 26 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1200, 1300, 2500, 3000, and 3200) are alsoapplicable in an analogous manner to method 2600 described above withrespect to FIG. 26. For example, the inputs, alert conditions, events,applications, states, and haptic outputs described above with referenceto method 2600 optionally have one or more of the characteristics of theinputs, alert conditions, events, applications, states, and hapticoutputs described herein with reference to other methods describedherein (e.g., methods 700, 900, 1000, 1100, 1200, 1300, 2500, 3000, and3200). For brevity, these details are not repeated here.

The operations described with reference to FIGS. 25-27 are optionallyimplemented by components depicted in FIGS. 28-29. FIG. 28 shows anexample functional block diagram of device 100 configured in accordancewith the principles of the various described embodiments. It isunderstood by persons of skill in the art that the functional blocksdescribed in FIG. 28 are, optionally, combined or separated intosub-blocks to implement the principles of the various describedembodiments. Therefore, the description herein optionally supports anypossible combination or separation or further definition of thefunctional blocks described herein.

As shown in FIG. 28, the device 100 includes a display unit 2801configured to display an application, a touch-sensitive surface unit2803 configured to receive user contacts, and a processing unit 2805coupled to the display unit 2801 and the touch-sensitive surface unit2803. In some embodiments, the processing unit 2805 includes a detectingunit 2810 and an alert providing unit 2815.

The following 4 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 28:

The processing unit 2805 is configured to detect an event at a devicewith a tactile output generator (e.g., with the detecting unit 2810),and responsive to detecting the event, in accordance with adetermination that the event is a type of event in a first event classthat includes a plurality of different event types, provide a firstalert that includes a first haptic component generated by the tactileoutput generator (e.g., with the alert providing unit 2815).

In accordance with a determination that the event is a type of event ina second event class that includes a plurality of different event types,processing unit 2805 is configured to provide a second alert thatincludes a second haptic component, generated by the tactile outputgenerator, that is different from the first haptic component (e.g., withthe alert providing unit 2815). In accordance with a determination thatthe event is a type of event in a third event class that includes aplurality of different event types, processing unit 2805 is configuredto provide a third alert that includes a third haptic component,generated by the tactile output generator, that is different from thefirst haptic component and the second haptic component (e.g., with thealert providing unit 2815).

According to some embodiments, the first class of events includes eventsgenerated based on communications from other users, the second class ofevents includes events generated based on predefined criteria entered bya user of the device, and the third class of events includes eventsgenerated based on the satisfaction of device-monitored criteria.According to some embodiments, the first haptic component is differentfrom the second haptic component and the third haptic component based onat least one of: spacing between haptic outputs, number of hapticoutputs, and change in the amplitude of haptic outputs over time duringthe haptic component (e.g., outputs provide by the alert providing unit2815).

According to some embodiments, the device has a plurality of uniquealerts that correspond to different events (e.g., provided by the alertproviding unit 2815), and the first event class, the second event class,and the third event class correspond to types of events that typicallyoccur more frequently than the events that correspond to unique alerts.

In response to detecting the event, in accordance with a determinationthat the event is a type of event in a fourth event class that includesa plurality of different event types, processing unit 2805 is configuredto provide a fourth alert that includes a fourth haptic component,generated by the tactile output generator, that is different from thefirst haptic component, the second haptic component, and the thirdhaptic component (e.g., with the alert providing unit 2815). Accordingto some embodiments, the fourth event class corresponds to an eventgenerated by a third party application. According to some embodiments,the fourth haptic component is differentiated from the first hapticcomponent, the second haptic component, and the third haptic componentby a haptic output element that is not included in the first hapticcomponent, the second haptic component, or the third haptic component.

As described above, the operations described with reference to FIGS.25-27 are optionally implemented by components depicted in FIGS. 28-29.FIG. 29 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 29 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 29, the device 100 includes a display unit 2901configured to display an application, a touch-sensitive surface unit2903 configured to receive user contacts, and a processing unit 2905coupled to the display unit 2901 and the touch-sensitive surface unit2903. In some embodiments, the processing unit 2905 includes a detectingunit 2910, an alert providing unit 2915, an alert-salience determinationunit 2920, and an setting change unit 2925.

The following 9 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 34:

The processing unit 2905 is configured to detect occurrence of a firstevent at a computing device with a tactile output generator (e.g., withthe detecting unit 2910), and in accordance with a determination thatthe first event is a first type of event, provide a first alert with thetactile output generator that includes a first haptic component selectedbased at least in part on an alert-salience setting of the device (e.g.,with the alert providing unit 2915). In accordance with a determinationthat the first event is a second type of event, processing unit 2905 isconfigured to provide a second alert with the tactile output generatorthat includes a second haptic component selected without regard to thealert-salience setting of the device (e.g., with the alert providingunit 2915). According to some embodiments, the first alert is differentfrom the second alert.

According to some embodiments, providing the first alert that includesthe haptic component selected based at least in part on thealert-salience setting of the device includes determining a value of thealert-salience setting (e.g., with the alert-salience determination unit2920). According to some embodiments, in accordance with a determinationthat the alert-salience setting has a first value, the first alert(e.g., provided by the alert providing unit 2915) includes a firsthaptic component and a second haptic component. According to someembodiments, in accordance with a determination that the alert-saliencesetting has a second value different from the first value, the firstalert (e.g., provided by the alert providing unit 2915) includes thefirst haptic component but does not include the second haptic component.According to some embodiments, the first value of the alert-saliencesetting is on and the second value of the alert-salience setting is off.According to some embodiments, the second haptic component precedes thefirst haptic component.

In accordance with a determination that the first event is a third typeof event, processing unit 2905 is configured to provide a third alertthat includes a haptic component selected based at least in part on thealert-salience setting of the device (e.g., provided by the alertproviding unit 2915). According to some embodiments, the processing unit2905 providing the third alert that includes the haptic componentselected based at least in part on the alert-salience setting of thedevice includes the processing unit 2905 being configured to determine avalue of the alert-salience setting (e.g., with the alert-saliencedetermination unit 2920).

According to some embodiments, in accordance with a determination thatthe alert-salience setting has the first value, the third alert includesa third haptic component and a fourth haptic component. According tosome embodiments, in accordance with a determination that thealert-salience setting has the second value different from the firstvalue, the third alert includes the third haptic component but does notinclude the fourth haptic component. According to some embodiments, thefourth haptic component is the same as the second haptic component.

According to some embodiments, when the first alert is detected whilethe alert-salience setting of the device has the first value, the firstalert provided in response to the first event includes the first hapticcomponent and the second haptic component and the second alert providedin response to the first event includes a second-alert haptic component(e.g., the alerts are provided by the alert providing unit 2915).According to some embodiments, while the alert-salience setting of thedevice has the first value, processing unit 2905 is configured toreceive a request to change the alert-salience setting of the device tothe second value (e.g., with the setting change unit 2925). Alsoaccording to some embodiments, in response to receiving the request tochange the alert-salience setting to the second value, processing unit2905 is configured to change the alert-salience setting to the secondvalue (e.g., with the setting change unit 2925).

According to some embodiments, while the alert-salience setting of thedevice has the second value, processing unit 2905 is configured todetect the occurrence of a second event (e.g., with the detecting unit2910) and in response to detecting the occurrence of the second event,in accordance with a determination that the second event is the firsttype of event, processing unit 2905 is configured to provide a thirdalert that includes the first haptic component and does not include thesecond haptic component (e.g., with the alert providing unit 2915).According to some embodiments, in accordance with a determination thatthe second event is the second type of event, processing unit 2905 isconfigured to provide the second alert that includes the second-alerthaptic component (e.g., with the alert providing unit 2915).

According to some embodiments, the processing unit 2905 is configured toprovide an additional haptic component for alerts triggered in responseto events in a first plurality of event classes when the alert-saliencesetting has a first value (e.g., with the alert providing unit 2915).According to some embodiments, the processing unit 2905 is configured toforgo providing the additional haptic component for alerts triggered inresponse to events in the first plurality of event classes when thealert-salience setting has the second value (e.g., with the alertproviding unit 2915). According to some embodiments, the processing unit2905 is configured to forgo providing the additional haptic componentfor alerts triggered in response to events in a second plurality ofevent classes, distinct from the first plurality of event classes,without regard to the alert-salience setting has the first value or thesecond value (e.g., with the alert providing unit 2915).

According to some embodiments, the first class of events includes eventsgenerated based on communications from other users, the second class ofevents includes events generated based on predefined criteria entered bya user of the device, and the third class of events includes eventsgenerated based on the satisfaction of device-monitored criteria.According to some embodiments, the first haptic component is differentfrom the second haptic component and the third haptic component based onat least one of: spacing between haptic outputs, number of hapticoutputs, and change in the amplitude of haptic outputs over time duringthe haptic component (e.g., outputs provide by the alert providing unit2915).

According to some embodiments, the device has a plurality of uniquealerts that correspond to different events (e.g., provided by the alertproviding unit 2915), and the first event class, the second event class,and the third event class correspond to types of events that typicallyoccur more frequently than the events that correspond to unique alerts.

Application-Specific Haptic/Audio Output

Alert conditions can be triggered within a variety of differentapplication classes (e.g., first party applications, third partyapplications), triggered by various applications (e.g., email, calendar,text message, payment application, timer application, etc.), andtriggered in many different contexts (e.g., action success, actionfailure, etc.). Which application class, which application, and/or inwhat context a detected condition is detected provides the basis forgenerating a corresponding output, according to some embodiments.Providing haptic and/or audio output that correlates with theapplication class, application, and/or context allows for consistency inhaptic and audio branding for the output, and thus creates a moreefficient human-machine interface, thereby reducing the amount of timeit takes for a user to perform operations, which consequently reducesenergy use and increases battery life for battery powered devices.

Table 12 provides an overview of various application classes,applications, and contexts as examples of output brand consistency usingboth haptic and audio output.

TABLE 12 OUTPUT SEMANTICS OVERVIEW Application Haptic Audio ClassApplications Condition Output Output First Party ANY ANY First hapticFirst class waveforms (first party) Third Party ANY ANY Second hapticSecond class waveforms (third party) First Party First ANY First hapticFirst application application waveforms First Party Second ANY Firsthaptic Second application application waveforms ANY Context 1 SuccessFirst (success) Context 1 haptic waveforms ANY Context 2 Success First(success) Context 2 haptic waveforms ANY Context 1 Failure SecondContext 1 (failure) haptic waveforms ANY Context 2 Failure SecondContext 2 (failure) haptic waveforms

The first two rows of Table 12 show a first application class, firstparty applications, and a second application class, third partyapplications. For applications within each class, the alert is a hapticoutput component corresponding to the class of applications combinedwith audio output component including waveform variants specific to thatapplication class, e.g., first party application alerts include a firsthaptic component and a variation of a first class waveform, whereasthird party application alerts include a second haptic component and avariation of a second class waveform. This maintains a consistent hapticassociated with each application class. However, first partyapplications each typically have their own haptic, while all third partyapplications typically share a single haptic. More specific examples arediscussed below in conjunction with Table 13.

The third and fourth rows of Table 12 show two different applicationswithin a single class (first party applications), based on anytriggering condition. Both applications result in the same (first)haptic output, which corresponds to the first party application class,but each has a different audio output corresponding to a waveformvariant for a waveform associated with the particular application. Forexample, the first application pairs the first haptic output with awaveform variant associated with the first application, with differentvariants for different conditions within the first application.Similarly, the second application pairs the first haptic output with awaveform variant associated with the second application, with differentvariants for different conditions within the second application. Thismaintains a consistent haptic associated with each application. Morespecific examples are discussed below in conjunction with Table 14.

The last four rows of Table 12 show two different conditions (success,failure), with haptic output consistent across the condition and audiooutput variable based on the specific context. For example, a successcondition provides a consistent first haptic associated with success,whereas context 1 provides a first set of waveforms associated withcontext 1 and context 2 provides a second set of waveforms associatedwith context 2. Likewise, a failure condition provides a consistentfirst haptic associated with failure, whereas context 1 provides a firstset of waveforms associated with context 1 and context 2 provides asecond set of waveforms associated with context 2. This maintains aconsistent haptic associated with each condition (success, failure).More specific examples are discussed below in conjunction with Table 15.

Which application class an alert condition is triggered within providesthe basis for generating a corresponding output, according to someembodiments. FIG. 30 is a flow diagram of a method 3000 for detecting afirst condition and a second condition and generating correspondingalerts including haptic and audio outputs according to some embodiments.Note that in some embodiments, different steps may be performed otherthan those shown in FIG. 30.

The method begins by, at an electronic device with one or more hapticoutput devices and one or more audio output devices, detecting 3005occurrence of a first condition at the device. The first condition canbe any of the various alert conditions described elsewhere herein, andthe application can be any application executing on the device 100,according to some embodiments. For example, in some embodiments, theoccurrence of the condition includes one or more of the receipt of acommunication (e.g., a message from an external source), the detectionof an input from a user (e.g., a touch, voice, or mechanical input thatis associated with feedback), or the detection of the satisfaction oftriggering criteria (e.g., the expiration of an alarm, as the completionof a workout, or the like), to name a few. According to someembodiments, the computing device 100 comprises a touch-sensitivedisplay for receiving the user input.

Responsive to detecting the occurrence of the first condition, thedevice 100 generates 3010 a first alert that corresponds to a respectiveapplication in a first class of applications. The first alert includes,in this example, a first haptic component 3012 that is output via theone or more haptic output devices and a first audio component 3014,output via the one or more audio output devices. In this instance, theaudio component is composed from an audio waveform that is designatedfor use by the respective application in the first class ofapplications. In some embodiments, the first class of applicationsincludes first party applications and the audio component is a uniqueaudio sample that is used to distinguish alerts for the respectiveapplication from alerts for other applications. According to someembodiments, the first haptic output is provided via the touch-sensitivedisplay on the computing device 100.

The device 100 also detects 3015 occurrence of a second condition at thedevice 100, optionally at a later time. Responsive to detecting theoccurrence of the second condition, the device generates 3020 a secondalert that corresponds to a respective application in a second class ofapplications that is different from the first class of applications, thesecond alert including a second haptic component 3022 that is output viathe one or more haptic output devices and a second audio component 3024,output via the one or more audio output devices. In this instance, theaudio component is composed from an audio waveform that is designatedfor use by applications in the second class of applications. In someembodiments, the second class of applications includes third partyapplications and the audio component is an audio sample that is used todistinguish alerts for third party applications from alerts for firstparty applications. According to some embodiments, the first hapticoutput is provided via the touch-sensitive display on the computingdevice 100.

Table 13 shows exemplary haptic and audio outputs corresponding toalerts for various application classes. The first three rows of thetable show alerts corresponding to a first application class, e.g.,first party applications. For each of the first party email, calendar,and text message applications, the output includes a first haptic thatcorresponds to the first application class (first party applications)and an audio output that includes a waveform instance corresponding tothe first application class (first party applications). Waveformvariants are discussed in greater detail below in conjunction with FIG.31. By providing haptic and audio output that correlates with theapplication class, the user is provided consistency in branding foroutput associated with the application class. Tables 9 and 10 above alsoprovide examples of both first party and third party alerts.

Similarly, the last three rows of the table show alerts corresponding toa second application class, e.g., third party applications. For each ofthe third party message, notification, and alert, the output includes asecond haptic that corresponds to the second application class (thirdparty applications), which is different from the first hapticcorresponding to the first party application according to someembodiments, and an audio output that includes a waveform instancecorresponding to the second application class (third partyapplications). By providing haptic and audio output that correlates withthe application class, the user is provided consistency in branding foroutput associated with the application class.

TABLE 13 APPLICATION CLASS OUTPUTS Application Haptic Audio ClassApplication Condition Output Output First Party Email Email First hapticFirst class received Waveform 1 First Party Calendar Calendar Firsthaptic First class notification Waveform 2 First Party Text Text messageFirst haptic First class message received waveform 3 Third Party 3Pmessage Message Second Second class application received haptic waveform1 Third Party 3P Notification Second Second class notification eventhaptic waveform 2 Third Party 3P alert Alert received Second Secondclass haptic waveform 3

In addition, although the third party applications have a unique set ofhaptic alerts, they also can share some of the same haptic feedback. Forexample, an alert that simulates physical feedback, such as haptic/audioclick or a haptic that provides simulated bouncing (rubber band-like)effect that is provided in conjunction with a user interface elementsuch as a scrollable list view bouncing when it reaches the end of thedisplay might be available regardless of the class of application inuse. If the application includes a scrollable list, then the output isavailable for use within the application upon that action, regardless ofthe application class.

According to some embodiments, the generating 3010 of the first alert ofthe method is in accordance with a determination that the alertcorresponds to a respective application in a first class ofapplications, and the generating 3020 of the second alert is inaccordance with a determination that the alert corresponds to arespective application in a second class of applications that isdifferent from the first class of applications.

According to some embodiments, the first haptic component and the secondhaptic component are composed from one or more haptic waveforms that areavailable to both the first class of application and the second class ofapplication. For example, any of the various waveforms described herein,e.g., minitap, microtap, tap, sine wave, etc. FIG. 5, described above,depicts examples of various haptic waveform morphologies.

Device 100 is configured to generate 3010, 3020 different alertsaccording to some embodiments. The different alerts include a firstsubset of two or more alerts that include audio components that areavailable to the first class of applications and are not available tothe second class of applications. For example, certain audio componentsmay be designated for use by first party applications, and not availablefor use by third party applications. The different alerts also include asecond subset of two or more alerts that include audio components thatare available to the second class of applications and are composed fromthe audio waveform that is designated for use by the second class ofapplications. For example, a particular audio waveform may be specificfor use by third party applications, and thus provide a consistent audiocharacteristic of third party applications. FIG. 6, described above,depicts examples of various audio waveform morphologies.

According to some embodiments, the second subset of alerts include twoalerts generated by applying different transformations to the samerespective audio sample (e.g., a natural audio sample that is arecording of a sound with a microphone or a synthetic audio sample thatwas generated digitally) that is associated with the second class ofapplications. In some embodiments, the two alerts include an alertincluding an audio component generated by applying a firsttransformation the respective audio sample that is associated with thesecond class of applications. In some embodiments, this alert isavailable to a plurality of applications in the second class ofapplications. The two alerts also, optionally, include a different alertincluding an audio component generated by applying a secondtransformation to the respective audio sample that is associated withthe second class of applications, wherein the first transformation isdifferent from the second transformation. In some embodiments, thisdifferent alert is available to a plurality of applications in thesecond class of applications. The transformations applied to thewaveforms include changing the amplitude, duration, pitch, or number ofrepetitions of the audio sample, for example.

Likewise, according to some embodiments the first subset of alertsinclude two alerts generated by applying different transformations to a(same) respective audio sample that is associated with a firstapplication in the first class of applications. In some embodiments, thetwo alerts include an alert including an audio component generated byapplying a first transformation the respective audio sample that isassociated with the first application in the first class of applications(e.g., by changing an amplitude, duration, pitch, or number ofrepetitions of the audio sample). In some embodiments, this alert isavailable to a plurality of applications in the first class ofapplications. In some embodiments, this alert is available only to thefirst application in the first class of applications. The two alertsalso, optionally, include a different alert including an audio componentgenerated by applying a second transformation to the respective audiosample that is associated with the first application in the first classof applications, wherein the first transformation is different from thesecond transformation. In some embodiments, this different alert isavailable to a plurality of applications in the first class ofapplications. In some embodiments, this different alert is availableonly to the first application in the first class of applications.

Similarly, according to some embodiments, the first subset of alertsinclude two alerts generated by applying different transformations tothe same respective audio sample that is associated with a secondapplication in the first class of applications, wherein the respectiveaudio sample that is associated with the second application is differentfrom the respective audio sample that is associated with the firstapplication. For example, the first application and the secondapplication each have characteristic sounds that serve as a “brand” forthe application, e.g., a payment application uses alerts generated witha sound sample that is unique to the payment application so that paymentsuccess, payment failure, and account activation for the paymentapplication have a recognizably similar sound. By comparison, a virtualassistant application uses alerts generated with a sound sample that isunique to the virtual assistant application, so that virtual assistantsuccess, virtual assistant failure, and virtual assistant ready all havea recognizably similar sound that is different from the recognizablysimilar sound of alerts for the payment application. In someembodiments, the two alerts include an alert including an audiocomponent generated by applying a first transformation the respectiveaudio sample that is associated with the second application in the firstclass of applications (e.g., by changing an amplitude, duration, pitch,or number of repetitions of the audio sample). In some embodiments, thisalert is available to a plurality of applications in the first class ofapplications. In some embodiments, this alert is available only to thesecond application in the first class of applications. The two alertsalso, optionally, include a different alert including an audio componentgenerated by applying a second transformation to the respective audiosample that is associated with the second application in the first classof applications, wherein the first transformation is different from thesecond transformation. In some embodiments, this different alert isavailable to a plurality of applications in the first class ofapplications. In some embodiments, this different alert is availableonly to the second application in the first class of applications.

Which application an alert condition is triggered within provides thebasis for generating a corresponding output, according to someembodiments. Table 14 shows exemplary haptic and audio outputscorresponding to alerts for two different applications (email, payment)within a single application class (first party applications). The firstthree rows of the table show alerts corresponding to a firstapplication, e.g., email. For each of the email conditions emailreceived, email sent, and email deleted, the output includes a firsthaptic that corresponds to the first application class (first partyapplications), and an audio output that includes a waveform instancecorresponding to the email application. Each email application conditionhas a waveform variant of the email waveform: first application waveform1, 2, 3. Waveform variants are discussed in greater detail below inconjunction with FIG. 31. By providing audio output that correlates withthe email application, the user is provided consistency in branding foroutput associated with the email application.

The last three rows of the table show alerts corresponding to a secondapplication, e.g., payment. For each of the payment conditions paymentsuccess, payment failure, account activation, the output includes afirst haptic that corresponds to the first application class (firstparty applications), and an audio output that includes a waveforminstance corresponding to the payment application. Each paymentapplication condition has a waveform variant of the payment waveform:second application waveform 1, 2, 3. By providing audio output thatcorrelates with the payment application, the user is providedconsistency in branding for output associated with the paymentapplication.

TABLE 14 APPLICATION-SPECIFIC AUDIO FOR SHARED HAPTIC Application HapticAudio Class Application Condition Output Output First Party Email EmailFirst haptic First received application waveform 1 First Party EmailEmail sent First haptic First application waveform 2 First Party EmailEmail deleted First haptic First application waveform 3 First PartyPayment Payment First haptic Second success application waveform 1 FirstParty Payment Payment First haptic Second failure application waveform 2First Party Payment Account First haptic Second activation applicationwaveform 3

According to some embodiments, the second subset of alerts is notavailable to the first class of applications. For example, first partyapplications are not allowed to use the third party application alertsounds, and optionally vice versa. According to some embodiments, boththe first and second subsets of alerts include haptic components thatare available to the first class of applications and the second class ofapplications, such that both third party applications are allowed to useshared haptic components.

According to some embodiments, the different alerts include a thirdsubset of two or more alerts that include audio components that areavailable to the second class of applications and the first class ofapplications, e.g., they are designated for shared use. For example, analert that simulates physical feedback, such as haptic/audio click or ahaptic that provides simulated bouncing (rubber band-like) effect thatis provided in conjunction with a user interface element such as ascrollable list view bouncing when it reaches the end of the displaymight be available regardless of the class of application in use. If theapplication includes a scrollable list, then the output is available foruse within the application upon that action, regardless of theapplication class. Thus, although third party applications have a uniqueset of haptic alerts, as noted in conjunction with Table 13 above, theyalso can share some of the same haptic feedback.

According to some embodiments, the first audio component, output via theone or more audio output devices, is composed from an audio waveformthat is designated for use by a first application (e.g., a paymentapplication) in the first class of applications (e.g., first partyapplications). For example, the audio component optionally is a uniqueaudio sample that is used to distinguish alerts for the firstapplication from alerts for other applications. According to someembodiments, the first audio component, output via the one or more audiooutput devices, is composed from an audio waveform that is designatedfor use by a second application (e.g., a virtual assistant application)in the first class of applications (e.g., first party applications). Forexample, the audio component optionally is a unique audio sample that isused to distinguish alerts for the second application from alerts forother applications.

According to some embodiments, the second audio component, output viathe one or more audio output devices, is composed from the audiowaveform that is designated for use by applications in the second classof application when the alert corresponds to a third application (e.g.,a payment application) in the second class of applications (e.g., firstparty applications). For example, the audio component optionally is aunique audio sample that is used to distinguish alerts for the firstapplication from alerts for other applications.

According to some embodiments, the first audio component, output via theone or more audio output devices, is composed from the audio waveformthat is designated for use by applications in the second class ofapplication when the alert corresponds to a second application (e.g., avirtual assistant application) in the second class of applications(e.g., first party applications). For example, the audio component is aunique audio sample that is used to distinguish alerts for the firstapplication from alerts for other applications.

It should be understood that the particular order in which theoperations in FIG. 30 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1200, 1300, 2500, 2600, and 3200) are alsoapplicable in an analogous manner to method 3000 described above withrespect to FIG. 30. For example, the inputs, alert conditions, events,applications, states, and haptic outputs described above with referenceto method 3000 optionally have one or more of the characteristics of theinputs, alert conditions, events, applications, states, and hapticoutputs described herein with reference to other methods describedherein (e.g., methods 700, 900, 1000, 1100, 1200, 1300, 2500, 2600, and3200). For brevity, these details are not repeated here.

FIG. 31 depicts various instances of a waveform according to someembodiments. The waveform can be haptic or audio. As shown, the waveform3105 has certain amplitudes 3110 associated with its features, as wellas an overall duration 3115 and a frequency 3120 between features of thewaveform. Waveform 3125 is waveform 3105 with a transformation appliedthat does not affect the amplitudes 3110, but results in a longeroverall duration 3130 and lower frequency 3135 between features of thewaveform. Waveform 3140 is waveform 3105 with a transformation appliedthat affects both the amplitudes 3145, and results in a shorter overallduration 3150 and higher frequency 3155 between features of thewaveform.

Relating these transformed waveforms to the discussion above, waveform3105 is designated for use with a certain application class orapplication. For example, a payment application is associated with audiowaveform 3105 according to some embodiments. In this example, alerts aregenerated with a sound sample that is unique to the payment applicationsuch that waveform instance 3125 indicates that payment success,waveform instance 3140 indicates payment failure, and waveform instance3105 indicates account activation for the payment application. Becauseeach waveform instance 3105, 3125, 3140 are based on the same soundsample of waveform 3105, all three have a recognizably similar soundthat the user will come to associate with the payment application.

In what context an alert condition is triggered provides the basis forgenerating a corresponding output, according to some embodiments.Providing haptic and/or audio output that correlates with the contextallows for consistency in haptic and audio branding for the output, andthus creates a more efficient human-machine interface, thereby reducingthe amount of time it takes for a user to perform operations, whichconsequently reduces energy use and increases battery life for batterypowered devices. FIG. 32 is a flow diagram of a method 3200 fordetecting a first condition in a first and second context and generatingcorresponding alerts including haptic and audio outputs according tosome embodiments. Note that in some embodiments, different steps may beperformed other than those shown in FIG. 32.

The method begins by, at an electronic device 100 with one or morehaptic output devices and one or more audio output devices, detecting3205 occurrence of a first condition in a first context at the device.For example, the condition is a failure condition and the context of thefailure event is that it occurred in a first application. According tosome embodiments, the occurrence of the condition includes one or moreof the receipt of a communication (e.g., a message from an externalsource), the detection of an input from a user (e.g., a touch, voice, ormechanical input that is associated with feedback), and the detection ofthe satisfaction of triggering criteria (e.g., the expiration of analarm, as the completion of a workout, or the like), to name a few.According to some embodiments, the computing device 100 comprises atouch-sensitive display for receiving the user input.

Responsive to detecting the occurrence of the first condition in thefirst context, the device 100 generates 3210 a first alert correspondingto the first condition that includes a first haptic component 3212 thatindicates the type of condition that occurred (e.g., a failure eventoccurred) and a first audio component 3214 that indicates that the firstcondition occurred in the first context (e.g., within a particularapplication. According to some embodiments, the first haptic output isprovided via the touch-sensitive display on the computing device 100.

The device 100 also detects 3215 occurrence of the first condition in asecond context that is different from the first context (e.g., in asecond application different from the first application. Response todetecting the occurrence of the first condition in the second context,the device 100 generates 3220 a second alert corresponding to the firstcondition that includes the first haptic component 3212 that indicatesthat the first condition occurred, e.g., the same haptic componentcorresponding to a failure in the second application context, and asecond audio component 3222, different from the first audio component,that indicates that the first condition occurred in the second contextthat is different from the first context. For example, the failure eventoccurred in a second application that is different from the firstapplication, or the failure event includes the option for the user tosubmit another request for performance of the operation that failed, andthus the audio portion of the output is different from that of the sameconditions (failure) in the first context.

According to some embodiments, the generating 3210 of the first alert ofthe method is in accordance with a determination that the alertcorresponds to the first condition having occurred in a first context,and the generating 3220 of the second alert is in accordance with adetermination that the first condition occurred in a second context.

By keeping a consistent haptic for a condition (e.g., failure), thehaptic waveform has a semantic meaning. Similarly, the haptic meaningassociated with the condition is independent of the semantic meaning ofthe audio component. For example, the audio alert provides context forthe haptic alert according to some embodiments because there is agreater dynamic range for audio than for haptic.

According to some embodiments, the method further includes detecting3230 the occurrence of the first condition (e.g., failure) at the devicein a third context different from the first and second contexts.Responsive to detecting the occurrence of the first condition at thedevice in the third context, the device 100 generates 3240 a third alertcorresponding to the first condition in the third context that includesthe first haptic component 3212 that indicates the type of conditionthat occurred (e.g., the same haptic component indicating that a failureevent occurred) and a third audio component 3242, different from thefirst audio component and the second audio component, that indicatesthat the first condition occurred in the third context (e.g., thefailure event occurred in a third application that is different from thefirst application). According to some embodiments, the generating 3240of the third alert of the method is in accordance with a determinationthat the alert corresponds to the first condition having occurred in athird context.

The context in which an alert condition is triggered provides the basisfor generating a corresponding output, according to some embodiments.Table 13 shows exemplary haptic and audio outputs corresponding toalerts for two different conditions (success, failure), in differentcontexts, and across two different application classes. The first tworows of the table show alerts corresponding to a first condition(success). For each context of a success (payment application, emailapplication), the output includes a first haptic that corresponds to thecondition (success), and an audio output that includes a waveforminstance corresponding to the context of the success (payment, messagesend).

The third and fourth rows of the table show alerts corresponding to asecond condition (failure). For each context of a failure (paymentapplication, email application), the output includes a second hapticthat corresponds to the condition (failure), and an audio output thatincludes a waveform instance corresponding to the context of the success(payment, message send).

The first four rows also correspond to a first application class, firstparty applications. The last two rows of the table show alertscorresponding to success and failure conditions in the context of asecond class of application, e.g., third party applications. In thisexample, the application is the same, but the audio output variesbecause the audio provided uses the application class as the context andprovides second class waveforms 1, 2 along with the corresponding hapticfor the condition (success failure). By providing haptic output thatconsistently correlates with the condition (success, failure) regardlessof context, the user is provided consistency in branding for outputassociated with the condition, and the audio alert provides context forthe haptic alert because there is a greater dynamic range available foraudio than for haptic.

TABLE 13 CONTEXT-SPECIFIC AUDIO FOR SHARED HAPTIC Application HapticAudio Class Application Condition Output Output First Party PaymentPayment First haptic Context success waveform 1 First Party EmailMessage send First haptic Context success waveform 2 First Party PaymentPayment Second Context failure haptic waveform 1 (First classwaveform 1) First Party Email Message send Second Context failure hapticwaveform 2 (First class waveform 2) Third Party Email Message send Firsthaptic Second class success waveform 1 Third Party Email Message sendSecond Second class failure haptic waveform 2

According to some embodiments, the first haptic component indicates thatan operation requested by the user has failed to occur, the first audiocomponent indicates that the operation failed to occur in a firstapplication (e.g., failure to make payment using a first party paymentapplication), and the second audio component indicates that theoperation failed to occur in a second application (e.g., failure to senda message in a third party messaging application). Thus, the hapticcomponent is shared and the audio component varies based on theapplication the failure occurred in.

According to some embodiments, the first haptic component indicates thatan operation requested by the user has successfully to occurred, thefirst audio component indicates that the operation occurred in a firstapplication and (e.g., a successful payment using a first party paymentapplication), and the second audio component indicates that theoperation occurred in a second application. (e.g., a successfully sentmessage in a third party messaging application). Thus, the hapticcomponent is shared and the audio component varies based on theapplication the success occurred in. According to some embodiments, thefirst audio component is generated based on an audio waveform that isdesignated for use by a first application. In some embodiments, theaudio waveform that is designated for use by the first application isalso used to generate other audio components for alerts that correspondto the first application, e.g., as described in greater detail above inconjunction with FIG. 30. The second audio component is generated basedon an audio waveform that is designated for use by a second applicationthat is different from the first application. In some embodiments, theaudio waveform that is designated for use by the second application isalso used to generate other audio components for alerts that correspondto the second application, e.g., as described in greater detail above inconjunction with FIG. 30.

According to some embodiments, the first audio component is generatedbased on an audio waveform that is designated for use by an applicationin a first class of applications. In some embodiments, the audiowaveform that is designated for use by a first party application is alsoused to generate other audio components for alerts that correspond tothe first party application, e.g., as described in greater detail abovein conjunction with FIG. 30. The second audio component is generatedbased on an audio waveform that is designated for use by applications ina second class of applications that is different from the first class ofapplications. In some embodiments, the audio waveform that is designatedfor use by third party applications is also used to generate other audiocomponents for alerts that correspond to the third party applications,e.g., as described in greater detail above in conjunction with FIG. 30.

According to some embodiments, the device is configured to providealerts that include a haptic component and an audio component inresponse to a plurality of different conditions and a plurality ofdifferent contexts. For example, the first condition is signified by thefirst haptic component, and the second condition that is different thanthe first condition is signified by the second haptic component that isdifferent than the first haptic component. In some embodiments, thefirst haptic component and the second haptic component are differentbecause they have a different duration, magnitude, or waveform. In someembodiments, the first audio component and the second audio componentare different because they include different numbers of sequentialhaptic outputs. According to some embodiments, the first context issignified by the first audio component, and the second context that isdifferent from the first context is signified by the second audiocomponent that is different from the first audio component. In someembodiments, the first audio component and the second audio componentare different because they are based on different audio samples. In someembodiments, the first audio component and the second audio componentare different because they include different numbers of sequentialnotes.

According to some embodiments, the first condition indicates success ofa requested operation and the first haptic component includes a seriesof haptic outputs that are aligned with features in the first audiocomponent. For example, the alignment may mean the features between thehaptic and audio outputs are in synch, mimic one another, and/or mirrorone another, e.g., separate haptic outputs in the haptic component aregenerated that correspond to sequential notes in the audio component.For example, haptic outputs are provided just before or during thesequential notes in the audio component, according to some embodiments.According to some embodiments, the first condition indicates failure ofa requested operation and the first haptic component is unaligned withfeatures in the first audio component, e.g., haptic outputs in thehaptic component are generated that do not correspond to sequentialnotes in the audio component. For example, a continuous haptic output isprovided that extends across a plurality of sequential notes in theaudio component.

According to some embodiments, the first condition indicates success ofa requested operation and the first haptic component includes a seriesof haptic outputs that are aligned with features in the second audiocomponent, e.g., separate haptic outputs in the haptic component aregenerated that correspond to sequential notes in the audio component.For example, haptic outputs are provided just before or during thesequential notes in the audio component according to some embodiments.According to some embodiments, the first condition indicates failure ofa requested operation, the first haptic component is unaligned withfeatures in the second audio component, e.g., haptic outputs in thehaptic component are generated that do not correspond to sequentialnotes in the audio component. For example, a continuous haptic output isprovided that extends across a plurality of sequential notes in theaudio component according to some embodiments.

As discussed in conjunction with FIG. 6, a respective haptic output maybe accompanied by a corresponding audio output, with at least a portionof the respective audio output concurrent with at least a portion of therespective haptic output, or the respective audio output occurs close intime to the respective haptic output so that the respective hapticoutput and the respective audio output are perceptually concurrent orsimultaneous. The haptic and audio waveforms need not be perfectlyaligned, and the device can take into account the fact that for certainclasses of haptic and audio outputs, the haptic and audio outputs willbe perceived by a user to be concurrent or simultaneous even if they areslightly offset in time (e.g., because audio output is processed morequickly than haptic output, so providing haptic output before providingaudio output will, in some circumstance, cause a user to perceive theaudio and haptic output as occurring concurrently or simultaneously).

It should be understood that the particular order in which theoperations in FIG. 32 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100, 1200, 1300, 2500, 2600, and 3000) are alsoapplicable in an analogous manner to method 3000 described above withrespect to FIG. 32. For example, the inputs, alert conditions, events,applications, states, and haptic outputs described above with referenceto method 3000 optionally have one or more of the characteristics of theinputs, alert conditions, events, applications, states, and hapticoutputs described herein with reference to other methods describedherein (e.g., methods 700, 900, 1000, 1100, 1200, 1300, 2500, 2600, and3000). For brevity, these details are not repeated here.

FIG. 33a shows various haptic and audio waveforms, both separate andconcurrent in time according to some embodiments. Haptic waveform 3305and audio waveform 3310, for example, can be used together as the outputfor an alert. As shown in combined output 3315, haptic waveform 3305 andaudio waveform 3310 are not aligned and will be perceived by the user asa haptic followed by audio followed by a haptic followed by audio.However, haptic waveform 3305 and audio waveform 3320, when usedtogether as shown in combined output 3325, are aligned, such that theaudio output is perceived by the user as concurrent with the hapticoutput, even though the alignment is not perfect.

FIG. 33b shows various haptic and audio waveforms, overlapping in time,according to some embodiments. Haptic waveform 3330, for example, is ahaptic waveform for a failure event according to some embodiments. Thefailure haptic 3330 is a sustained buzz according to some embodiments.When failure haptic 3330 is used in conjunction with different audiooutput, it signals different types of failures and/or failures indifferent contexts as described in conjunction with FIG. 32. Audiowaveform 3335 represents a third party audio waveform, in this example afailure audio sample that sounds like ding-ding-dun. As shown incombined output 3340, haptic waveform 3330 and audio waveform 3335 arealigned. Audio waveform 3345 represents a different third party audiowaveform, in this example a try again audio sample that sounds likedun-dun-dun. It is a similar audio sample to the failure audio waveform3335 as variations of a third party audio waveform, but has threesimilar features rather than two of one feature and third one that isdifferent. When combined with failure haptic 3330, combined waveform3350 results. Combined waveform 3350 shares the failure haptic 3330, butdiffers in context and thus audio output from audio waveform 3335.

FIG. 33c shows various haptic and audio waveforms, overlapping in time,according to some embodiments. Haptic waveform 3330, for example, is thesame haptic waveform for a failure event as FIG. 33b , according to someembodiments. Failure haptic 3330 is used in conjunction with twodifferent first party audio waveforms (3355, 3365) to produce twodifferent but relate first party alerts. Audio waveform 3355 representsa first party audio waveform for a payment failure audio sample thatsounds like de-dum, de-dum. Combined output 3360 shows haptic waveform3330 and audio waveform 3335 in alignment. Audio waveform 3365represents a different first party audio waveform, in this example apasscode entry failure sample that sounds like de-dum, de-dum, de-dum.When combined with failure haptic 3330, combined waveform 3370 results.Combined waveform 3370 shares the failure haptic 3330, but differs incontext and thus audio output from audio waveform 3355.

The operations described above with reference to FIGS. 30 and 32 are,optionally, implemented by components depicted in FIGS. 1A-1B, and34-35. FIG. 34 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 34 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 34, the device 100 includes a display unit 3401configured to display an application, a touch-sensitive surface unit3403 configured to receive user contacts, and a processing unit 3405coupled to the display unit 3401 and the touch-sensitive surface unit3403. In some embodiments, the processing unit 3405 includes a detectingunit 3410 and an alert generating unit 3415.

The following 9 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 34:

The processing unit 3405 is configured to detect, at an electronicdevice with one or more haptic output devices, occurrence of a firstcondition at the device (e.g., with the detecting unit 3410), andresponsive to detecting the occurrence, optionally in accordance with adetermination that the alert corresponds to a respective application ina first class of applications, generate a first alert that correspondsto a respective application in a first class of applications (e.g., withthe alert generating unit 3415). The processing unit 3405 is alsoconfigured to detect occurrence of a second condition at the device(e.g., with the detecting unit 3410), and in response generate a secondalert that corresponds to a respective application in a second class ofapplications that is different from the first class of applications, thesecond alert including a second haptic component that is output via theone or more haptic output devices and a second audio component, outputvia the one or more audio output devices, that is composed from an audiowaveform that is designated for use by applications in the second classof applications (e.g., with the alert generating unit 3415).

According to some embodiments, the first haptic component and the secondhaptic component are composed from one or more haptic waveforms that areavailable to both the first class of application and the second class ofapplication.

According to some embodiments, the device 100 is configured to generatea plurality of different alerts (e.g., by the alert generating unit3415). The plurality of different alerts includes a first subset of twoor more alerts that include audio components that are available to thefirst class of applications and are not available to the second class ofapplications and the plurality of different alerts includes a secondsubset of two or more alerts that include audio components that areavailable to the second class of applications and are composed from theaudio waveform that is designated for use by the second class ofapplications.

According to some embodiments, the second subset of alerts include twoalerts generated (e.g., by the alert generating unit 3415) by applyingdifferent transformations to a respective audio sample that isassociated with the second class of applications. According to someembodiments, the first subset of alerts include two alerts generated(e.g., by the alert generating unit 3415) by applying differenttransformations to a respective audio sample that is associated with afirst application in the first class of applications.

According to some embodiments, the first subset of alerts include twoalerts generated (e.g., by the alert generating unit 3415) by applyingdifferent transformations to a respective audio sample that isassociated with a second application in the first class of applications,wherein the respective audio sample that is associated with the secondapplication is different from the respective audio sample that isassociated with the first application. According to some embodiments,the second subset of alerts is not available to the first class ofapplications.

According to some embodiments, the first subset of two or more alerts(generated, e.g., by the alert generating unit 3415) include hapticcomponents that are available to the first class of applications and thesecond class of applications and the second subset of two or more alertsthat include haptic components that are available to the second class ofapplications and the first class of applications.

According to some embodiments, the plurality of different alerts(generated, e.g., by the alert generating unit 3415) includes a thirdsubset of two or more alerts that include audio components that areavailable to the second class of applications and the first class ofapplications.

According to some embodiments, the first audio component, output via theone or more audio output devices (e.g., by the alert generating unit3415), is composed from an audio waveform that is designated for use bya first application in the first class of applications and the firstaudio component, output via the one or more audio output devices, iscomposed from an audio waveform that is designated for use by a secondapplication in the first class of applications.

According to some embodiments, the second audio component, output viathe one or more audio output devices (e.g., by the alert generating unit3415), is composed from the audio waveform that is designated for use byapplications in the second class of application when the alertcorresponds to a third application in the second class of applicationsand the first audio component, output via the one or more audio outputdevices, is composed from the audio waveform that is designated for useby applications in the second class of application when the alertcorresponds to a second application in the second class of applications.

FIG. 35 shows an example functional block diagram of device 100configured in accordance with the principles of the various describedembodiments. It is understood by persons of skill in the art that thefunctional blocks described in FIG. 35 are, optionally, combined orseparated into sub-blocks to implement the principles of the variousdescribed embodiments. Therefore, the description herein optionallysupports any possible combination or separation or further definition ofthe functional blocks described herein.

As shown in FIG. 35, the device 100 includes a display unit 3501configured to display an application, a touch-sensitive surface unit3503 configured to receive user contacts, and a processing unit 3505coupled to the display unit 3501 and the touch-sensitive surface unit3503. In some embodiments, the processing unit 3505 includes a detectingunit 3510 and an alert generating unit 3515.

The following 10 paragraphs describe different embodiments that may beimplemented, individually or in any combination, by the device 100illustrated in FIG. 35:

The processing unit 3505 is configured to detect, at an electronicdevice with one or more haptic output devices, occurrence of a firstcondition in a first context (e.g., with the detecting unit 3510), andresponsive to detecting the occurrence, generate a first alertcorresponding to the first condition that includes a first hapticcomponent that indicates the type of condition that occurred and a firstaudio component that indicates that the first condition occurred in thefirst context (e.g., with the alert generating unit 3515).

The processing unit 3505 is also configured to detect occurrence at thedevice of the first condition in a second context that is different fromthe first context (e.g., with the detecting unit 3510), and responsiveto detecting the occurrence, generating a second alert corresponding tothe first condition that includes the first haptic component thatindicates that the first condition occurred and a second audiocomponent, different from the first audio component, that indicates thatthe first condition occurred in the second context that is differentfrom the first context (e.g., with the alert generating unit 3515).

The processing unit 3505 is also configured to detect occurrence at thedevice of the first condition at the device in a third context differentfrom the first and second contexts (e.g., with the detecting unit 3510),and responsive to detecting the occurrence of the first condition at thedevice in the third context, generating a third alert corresponding tothe first condition in the third context that includes the first hapticcomponent that indicates the type of condition that occurred and a thirdaudio component, different from the first audio component and the secondaudio component, that indicates that the first condition occurred in thesecond third context (e.g., with the alert generating unit 3515).

According to some embodiments, the first haptic component indicates thatan operation requested by the user has failed to occur, the first audiocomponent indicates that the operation failed to occur in a firstapplication, and the second audio component indicates that the operationfailed to occur in a second application.

According to some embodiments, the first haptic component indicates thatan operation requested by the user has successfully to occurred, thefirst audio component indicates that the operation occurred in a firstapplication; and the second audio component indicates that the operationoccurred in a second application.

According to some embodiments, the first audio component is generatedbased on an audio waveform that is designated for use by a firstapplication and the second audio component is generated based on anaudio waveform that is designated for use by a second application thatis different from the first application.

According to some embodiments, the first audio component is generatedbased on an audio waveform that is designated for use by an applicationin a first class of applications and the second audio component isgenerated based on an audio waveform that is designated for use byapplications in a second class of applications that is different fromthe first class of applications.

According to some embodiments, the device is configured to providealerts (e.g., with the alert generating unit 3515) that include a hapticcomponent and an audio component in response to a plurality of differentconditions and a plurality of different contexts, the first condition issignified by the first haptic component, the second condition that isdifferent than the first condition is signified by the second hapticcomponent that is different than the first haptic component, the firstcontext is signified by the first audio component, and the secondcontext that is different from the first context is signified by thesecond audio component that is different from the first audio component.

According to some embodiments, the first condition indicates success ofa requested operation and the first haptic component includes a seriesof haptic outputs that are aligned with features in the first audiocomponent and the first condition indicates failure of a requestedoperation and the first haptic component is unaligned with features inthe first audio component, (generated, e.g., by alert generating unit3515).

According to some embodiments, the first condition indicates success ofa requested operation and the first haptic component includes a seriesof haptic outputs that are aligned with features in the second audiocomponent and the first condition indicates failure of a requestedoperation, the first haptic component is unaligned with features in thesecond audio component.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a tangible computer readable storage medium or any typeof media suitable for storing electronic instructions, and coupled to acomputer system bus. Furthermore, any computing systems referred to inthe specification may include a single processor or may be architecturesemploying multiple processor designs for increased computing capability.

Embodiments of the invention may also relate to a computer data signalembodied in a carrier wave, where the computer data signal includes anyembodiment of a computer program product or other data combinationdescribed herein. The computer data signal is a product that ispresented in a tangible medium or carrier wave and modulated orotherwise encoded in the carrier wave, which is tangible, andtransmitted according to any suitable transmission method.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A method, comprising: at an electronic devicewith one or more haptic output devices and one or more audio outputdevices: detecting occurrence of a plurality of conditions in aplurality of software applications, wherein the electronic device isconfigured to: for conditions of a same type, generate respective alertsthat include same haptic components that indicate that the conditionsare of the same type, and for conditions of different types, generaterespective alerts that include different respective haptic componentsthat indicate that the conditions are of different types, regardless ofrespective software applications in the plurality of softwareapplications in which the conditions in the plurality of conditions weredetected; for conditions detected in a same software application in theplurality of software applications, generate respective alerts thatinclude same audio components that indicate that the conditions occurredin the same software application, and for conditions detected indifferent software applications in the plurality of softwareapplications, generate respective alerts that include different audiocomponents that indicate that the conditions occurred in differentsoftware applications, regardless of respective types of the conditions;wherein detecting the occurrence of the plurality of conditions in theplurality of software applications includes: detecting occurrence of afirst condition of a first type in a first software application at theelectronic device, wherein the first condition of the first typeindicates failure of an operation requested by a user to occur; inresponse to detecting the occurrence of the first condition in the firstsoftware application, generating a first alert corresponding to thefirst condition that includes: a first haptic component that indicatesthat the first condition of the first type has occurred; and a firstaudio component that corresponds to the first software application;detecting occurrence at the electronic device of the first condition ofthe first type in a second software application that is different fromthe first software application; in response to detecting the occurrenceof the first condition in the second software application, generating asecond alert corresponding to the first condition that includes: thefirst haptic component that indicates that the first condition of thefirst type has occurred; and a second audio component, different fromthe first audio component, the second audio component corresponding tothe second software application; detecting occurrence at the electronicdevice of a second condition of a second type in the first softwareapplication, the second type different than the first type; and inresponse to detecting the occurrence of the second condition in thefirst software application, generating a third alert corresponding tothe second condition that includes: a second haptic component thatindicates that the second condition of the second type has occurred, thesecond haptic component different than the first haptic component; andthe first audio component corresponding to the first softwareapplication.
 2. The method of claim 1, wherein detecting the occurrenceof the plurality of conditions in the plurality of software applicationsfurther comprises: detecting the occurrence of the first condition atthe electronic device in a third software application different from thefirst and second software applications, wherein the first conditionindicates failure of an operation in the third software applicationrequested by the user to occur; in response to detecting the occurrenceof the first condition at the electronic device in the third softwareapplication, generating a fourth alert corresponding to the firstcondition in the third software application that includes: the firsthaptic component that indicates that the first condition of the firsttype has occurred; and a third audio component, different from the firstaudio component and the second audio component, the third audiocomponent corresponding to the third software application.
 3. The methodof claim 1, wherein: the second haptic component in the third alertindicates that an operation in the first software application requestedby a user has successfully occurred.
 4. The method of claim 1, wherein:the first audio component is generated based on an audio waveform thatis designated for use by the first software application; and the secondaudio component is generated based on an audio waveform that isdesignated for use by the second software application.
 5. The method ofclaim 1, wherein: the first audio component is generated based on anaudio waveform that is designated for use by software applications in afirst class of software applications; and the second audio component isgenerated based on an audio waveform that is designated for use bysoftware applications in a second class of software applications that isdifferent from the first class of software applications.
 6. The methodof claim 1, wherein: the second condition indicates success of arequested operation, and the second haptic component includes a seriesof haptic outputs that are aligned with features in the first audiocomponent; and the first haptic component is unaligned with features inthe first audio component.
 7. The method of claim 1, including:detecting occurrence at the electronic device of the second condition ofthe second type in the second software application; and in response todetecting the occurrence of the second condition in the second softwareapplication, generating a fifth alert corresponding to the secondcondition that includes: the second haptic component that indicates thatthe second condition of the second type has occurred; and the secondaudio component that corresponds to the second software application;wherein: the second condition indicates success of a requestedoperation, and the second haptic component includes a series of hapticoutputs that are aligned with features in the second audio component;and the first haptic component is unaligned with features in the secondaudio component.
 8. The method of claim 1, wherein: the electronicdevice has a touch-sensitive display; and the first haptic component andthe second haptic component are provided via the touch-sensitive displayon the electronic device.
 9. A non-transitory computer readable storagemedium storing executable code, the code when executed causes aprocessor at a computing device with one or more haptic output devicesand one or more audio output devices to perform the following steps:detecting occurrence of a plurality of conditions in a plurality ofsoftware applications, wherein the computing device is configured to:for conditions of a same type, generate respective alerts that includesame haptic components that indicate that the conditions are of the sametype, and for conditions of different types, generate respective alertsthat include different respective haptic components that indicate thatthe conditions are of different types, regardless of respective softwareapplications in the plurality of software applications in which theconditions in the plurality of conditions were detected; for conditionsdetected in a same software application in the plurality of softwareapplications, generate respective alerts that include same audiocomponents that indicate that the conditions occurred in the samesoftware application, and for conditions detected in different softwareapplications in the plurality of software applications, generaterespective alerts that include different audio components that indicatethat the conditions occurred in different software applications,regardless of respective types of the conditions; wherein the code fordetecting the occurrence of the plurality of conditions in the pluralityof software applications includes instructions for: detecting occurrenceof a first condition of a first type in a first software application atthe computing device, wherein the first condition of the first typeindicates failure of an operation requested by a user to occur; inresponse to detecting the occurrence of the first condition in the firstsoftware application, generating a first alert corresponding to thefirst condition that includes: a first haptic component that indicatesthat the first condition of the first type has occurred; and a firstaudio component that corresponds to the first software application;detecting occurrence at the computing device of the first condition ofthe first type in a second software application that is different fromthe first software application; in response to detecting the occurrenceof the first condition in the second software application, generating asecond alert corresponding to the first condition that includes: thefirst haptic component that indicates that the first condition of thefirst type has occurred; and a second audio component, different fromthe first audio component, the second audio component corresponding tothe second software application; detecting occurrence at the computingdevice of a second condition of a second type in the first softwareapplication, the second type different than the first type; and inresponse to detecting the occurrence of the second condition in thefirst software application, generating a third alert corresponding tothe second condition that includes: a second haptic component thatindicates that the second condition of the second type has occurred, thesecond haptic component different than the first haptic component; andthe first audio component corresponding to the first softwareapplication.
 10. The non-transitory computer readable storage medium ofclaim 9, wherein the code for detecting the occurrence of the pluralityof conditions in the plurality of software applications further includesinstructions for: detecting the occurrence of the first condition at thecomputing device in a third software application different from thefirst and second software applications, wherein the first conditionindicates failure of an operation in the third software applicationrequested by the user to occur; in response to detecting the occurrenceof the first condition at the computing device in the third softwareapplication, generating a fourth alert corresponding to the firstcondition in the third software application that includes: the firsthaptic component that indicates that the first condition of the firsttype has occurred; and a third audio component, different from the firstaudio component and the second audio component, the third audiocomponent corresponding to the third software application.
 11. Thenon-transitory computer readable storage medium of claim 9, wherein: thesecond haptic component in the third alert indicates that an operationin the first software application requested by a user has successfullyoccurred.
 12. The non-transitory computer readable storage medium ofclaim 9, wherein: the first audio component is generated based on anaudio waveform that is designated for use by the first softwareapplication; and the second audio component is generated based on anaudio waveform that is designated for use by the second softwareapplication.
 13. The non-transitory computer readable storage medium ofclaim 9, wherein: the first audio component is generated based on anaudio waveform that is designated for use by software applications in afirst class of software applications; and the second audio component isgenerated based on an audio waveform that is designated for use bysoftware applications in a second class of software applications that isdifferent from the first class of software applications.
 14. Thenon-transitory computer readable storage medium of claim 9, wherein: thesecond condition indicates success of a requested operation, and thesecond haptic component includes a series of haptic outputs that arealigned with features in the first audio component; and the first hapticcomponent is unaligned with features in the first audio component. 15.The non-transitory computer readable storage medium of claim 9, whereinthe code for detecting the occurrence of the plurality of conditions inthe plurality of software applications further includes instructionsfor: detecting occurrence at the computing device of the secondcondition of the second type in the second software application; and inresponse to detecting the occurrence of the second condition in thesecond software application, generating a fifth alert corresponding tothe second condition that includes: the second haptic component thatindicates that the second condition of the second type has occurred; andthe second audio component that corresponds to the second softwareapplication; wherein: the second condition indicates success of arequested operation and the second haptic component includes a series ofhaptic outputs that are aligned with features in the second audiocomponent; and the first haptic component is unaligned with features inthe second audio component.
 16. The non-transitory computer readablestorage medium of claim 9, wherein: the computing device has atouch-sensitive display; and the first haptic component and the secondhaptic component are provided via the touch-sensitive display on thecomputing device.
 17. A device comprising: one or more haptic outputdevices; one or more audio output devices; a touch-sensitive surface;and memory storing one or more programs that include instructions for:detecting occurrence of a plurality of conditions in a plurality ofsoftware applications, wherein the device is configured to: forconditions of a same type, generate respective alerts that include samehaptic components that indicate that the conditions are of the sametype, and for conditions of different types, generate respective alertsthat include different respective haptic components that indicate thatthe conditions are of different types, regardless of respective softwareapplications in the plurality of software applications in which theconditions in the plurality of conditions were detected; and forconditions detected in a same software application in the plurality ofsoftware applications, generate respective alerts that include sameaudio components that indicate that the conditions occurred in the samesoftware application, and for conditions detected in different softwareapplications in the plurality of software applications, generaterespective alerts that include different audio components that indicatethat the conditions occurred in different software applications,regardless of respective types of the conditions; and wherein theinstructions for detecting the occurrence of the plurality of conditionsin the plurality of software applications include instructions for:detecting occurrence of a first condition of a first type in a firstsoftware application at the device, wherein the first condition of thefirst type indicates failure of an operation requested by a user tooccur; in response to detecting the occurrence of the first condition inthe first software application, generating a first alert correspondingto the first condition that includes: a first haptic component thatindicates that the first condition of the first type has occurred; and afirst audio component that corresponds to the first softwareapplication; detecting occurrence at the device of the first conditionof the first type in a second software application that is differentfrom the first software application; in response to detecting theoccurrence of the first condition in the second software application,generating a second alert corresponding to the first condition thatincludes: the first haptic component that indicates that the firstcondition of the first type has occurred; and a second audio component,different from the first audio component, the second audio componentcorresponding to the second software application; detecting occurrenceat the device of a second condition of a second type in the firstsoftware application, the second type different than the first type; andin response to detecting the occurrence of the second condition in thefirst software application, generating a third alert corresponding tothe second condition that includes: a second haptic component thatindicates that the second condition of the second type has occurred, thesecond haptic component different than the first haptic component; andthe first audio component corresponding to the first softwareapplication.
 18. The device of claim 17, wherein the instructions fordetecting the occurrence of the plurality of conditions in the pluralityof software applications further include instructions for: detecting theoccurrence of the first condition at the device in a third softwareapplication different from the first and second software applications,wherein the first condition indicates failure of an operation in thethird software application requested by the user to occur; in responseto detecting the occurrence of the first condition at the device in thethird software application, generating a fourth alert corresponding tothe first condition in the third software application that includes: thefirst haptic component that indicates that the first condition of thefirst type has occurred; and a third audio component, different from thefirst audio component and the second audio component, the third audiocomponent corresponding to the third software application.
 19. Thedevice of claim 17, wherein: the second haptic component in the thirdalert indicates that an operation in the first software applicationrequested by a user has successfully occurred.
 20. The device of claim17, wherein: the first audio component is generated based on an audiowaveform that is designated for use by the first software application;and the second audio component is generated based on an audio waveformthat is designated for use by the second software application.
 21. Thedevice of claim 17, wherein: the first audio component is generatedbased on an audio waveform that is designated for use by softwareapplications in a first class of software applications; and the secondaudio component is generated based on an audio waveform that isdesignated for use by software applications in a second class ofsoftware applications that is different from the first class of softwareapplications.
 22. The device of claim 17, wherein: the second conditionindicates success of a requested operation, and the second hapticcomponent includes a series of haptic outputs that are aligned withfeatures in the first audio component; and the first haptic component isunaligned with features in the first audio component.
 23. The device ofclaim 17, wherein the instructions for detecting the occurrence of theplurality of conditions in the plurality of software applicationsfurther include instructions for: detecting occurrence at the computingdevice of the second condition of the second type in the second softwareapplication; and in response to detecting the occurrence of the secondcondition in the second software application, generating a fifth alertcorresponding to the second condition that includes: the second hapticcomponent that indicates that the second condition of the second typehas occurred; and the second audio component that corresponds to thesecond software application; wherein: the second condition indicatessuccess of a requested operation and the second haptic componentincludes a series of haptic outputs that are aligned with features inthe second audio component; and the first haptic component is unalignedwith features in the second audio component.
 24. The device of claim 17,wherein: the first haptic component and the second haptic component aregenerated by the one or more haptic output devices and provided via thetouch-sensitive surface on the device.